Nov 04 04:45:14.773749 ip-10-0-134-224 systemd[1]: Starting Kubernetes Kubelet... Nov 04 04:45:15.234048 ip-10-0-134-224 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 04:45:15.234048 ip-10-0-134-224 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 04 04:45:15.234048 ip-10-0-134-224 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 04:45:15.234541 ip-10-0-134-224 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 04:45:15.237255 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237191 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:45:15.237297 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237256 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:45:15.237297 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237260 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:45:15.237297 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237263 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:45:15.237297 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237266 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:45:15.237297 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237269 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:45:15.237297 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237272 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:45:15.237297 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237275 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:45:15.237297 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237278 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:45:15.237297 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237281 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:45:15.237297 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237283 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:45:15.237297 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237286 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:45:15.237297 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237289 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:45:15.237297 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237292 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:45:15.237297 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237294 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:45:15.237297 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237297 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:45:15.237297 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237300 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:45:15.237297 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237303 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:45:15.237297 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237306 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:45:15.237766 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237308 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:45:15.237766 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237311 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:45:15.237766 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237314 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:45:15.237766 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237316 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:45:15.237766 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237319 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:45:15.237766 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237321 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:45:15.237766 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237324 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:45:15.237766 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237330 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:45:15.237766 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237333 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:45:15.237766 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237336 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:45:15.237766 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237338 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:45:15.237766 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237341 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:45:15.237766 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237343 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:45:15.237766 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237346 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:45:15.237766 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237349 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:45:15.237766 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237351 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:45:15.237766 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237354 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:45:15.237766 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237356 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:45:15.237766 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237359 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:45:15.237766 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237361 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:45:15.238225 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237364 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:45:15.238225 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237366 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:45:15.238225 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237369 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:45:15.238225 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237371 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:45:15.238225 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237374 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:45:15.238225 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237376 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:45:15.238225 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237379 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:45:15.238225 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237381 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:45:15.238225 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237385 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:45:15.238225 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237387 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:45:15.238225 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237390 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:45:15.238225 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237393 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:45:15.238225 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237395 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:45:15.238225 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237399 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:45:15.238225 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237401 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:45:15.238225 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237404 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:45:15.238225 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237407 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:45:15.238225 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237409 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:45:15.238225 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237412 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:45:15.238225 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237414 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:45:15.238706 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237417 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:45:15.238706 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237420 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:45:15.238706 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237423 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:45:15.238706 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237425 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:45:15.238706 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237428 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:45:15.238706 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237430 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:45:15.238706 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237433 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:45:15.238706 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237436 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:45:15.238706 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237438 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:45:15.238706 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237441 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:45:15.238706 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237444 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:45:15.238706 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237447 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:45:15.238706 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237449 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:45:15.238706 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237452 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:45:15.238706 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237454 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:45:15.238706 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237457 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:45:15.238706 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237459 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:45:15.238706 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237462 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:45:15.238706 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237464 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:45:15.239174 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237469 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:45:15.239174 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237472 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:45:15.239174 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237476 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:45:15.239174 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237480 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:45:15.239174 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237484 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:45:15.239174 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237487 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:45:15.239174 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237502 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:45:15.239174 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237507 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:45:15.239174 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237511 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:45:15.239174 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237514 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:45:15.239174 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237518 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:45:15.239174 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237521 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:45:15.239174 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237524 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:45:15.239174 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237528 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:45:15.239174 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237530 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:45:15.239174 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237533 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:45:15.239174 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237537 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:45:15.239174 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237539 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:45:15.239174 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237542 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:45:15.239638 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237545 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:45:15.239638 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237547 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:45:15.239638 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237550 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:45:15.239638 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237553 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:45:15.239638 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237556 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:45:15.239638 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237558 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:45:15.239638 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237561 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:45:15.239638 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237564 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:45:15.239638 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.237566 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:45:15.239638 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238328 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:45:15.239638 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238334 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:45:15.239638 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238337 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:45:15.239638 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238340 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:45:15.239638 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238343 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:45:15.239638 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238346 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:45:15.239638 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238349 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:45:15.239638 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238352 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:45:15.239638 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238355 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:45:15.239638 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238358 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:45:15.239638 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238361 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:45:15.240097 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238364 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:45:15.240097 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238367 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:45:15.240097 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238369 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:45:15.240097 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238372 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:45:15.240097 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238375 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:45:15.240097 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238378 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:45:15.240097 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238380 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:45:15.240097 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238383 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:45:15.240097 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238385 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:45:15.240097 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238388 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:45:15.240097 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238391 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:45:15.240097 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238395 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:45:15.240097 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238399 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:45:15.240097 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238403 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:45:15.240097 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238407 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:45:15.240097 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238410 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:45:15.240097 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238412 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:45:15.240097 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238415 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:45:15.240097 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238417 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:45:15.240560 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238420 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:45:15.240560 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238422 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:45:15.240560 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238425 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:45:15.240560 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238427 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:45:15.240560 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238430 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:45:15.240560 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238433 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:45:15.240560 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238435 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:45:15.240560 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238438 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:45:15.240560 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238440 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:45:15.240560 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238443 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:45:15.240560 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238446 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:45:15.240560 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238449 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:45:15.240560 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238453 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:45:15.240560 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238456 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:45:15.240560 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238459 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:45:15.240560 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238462 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:45:15.240560 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238464 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:45:15.240560 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238467 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:45:15.240560 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238471 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:45:15.241019 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238473 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:45:15.241019 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238476 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:45:15.241019 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238478 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:45:15.241019 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238481 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:45:15.241019 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238484 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:45:15.241019 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238486 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:45:15.241019 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238489 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:45:15.241019 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238506 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:45:15.241019 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238508 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:45:15.241019 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238511 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:45:15.241019 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238514 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:45:15.241019 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238517 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:45:15.241019 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238519 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:45:15.241019 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238522 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:45:15.241019 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238524 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:45:15.241019 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238527 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:45:15.241019 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238529 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:45:15.241019 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238532 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:45:15.241019 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238534 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:45:15.241019 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238537 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:45:15.241543 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238539 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:45:15.241543 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238542 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:45:15.241543 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238544 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:45:15.241543 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238547 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:45:15.241543 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238551 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:45:15.241543 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238553 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:45:15.241543 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238556 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:45:15.241543 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238559 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:45:15.241543 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238561 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:45:15.241543 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238564 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:45:15.241543 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238567 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:45:15.241543 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238569 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:45:15.241543 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238572 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:45:15.241543 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238574 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:45:15.241543 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238577 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:45:15.241543 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238579 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:45:15.241543 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238582 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:45:15.241543 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238585 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:45:15.241543 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238587 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:45:15.241999 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238590 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:45:15.241999 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238594 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:45:15.241999 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238598 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:45:15.241999 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238602 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:45:15.241999 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238605 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:45:15.241999 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238608 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:45:15.241999 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238611 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:45:15.241999 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238613 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:45:15.241999 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238616 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:45:15.241999 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238619 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:45:15.241999 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238621 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:45:15.241999 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238624 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:45:15.241999 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238626 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:45:15.241999 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238629 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:45:15.241999 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238631 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:45:15.241999 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238633 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:45:15.241999 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238637 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:45:15.241999 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.238641 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:45:15.241999 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239123 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:45:15.242434 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239128 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:45:15.242434 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239131 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:45:15.242434 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239134 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:45:15.242434 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239136 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:45:15.242434 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239139 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:45:15.242434 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239141 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:45:15.242434 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239144 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:45:15.242434 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239147 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:45:15.242434 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239150 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:45:15.242434 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239152 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:45:15.242434 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239155 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:45:15.242434 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239158 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:45:15.242434 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239160 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:45:15.242434 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239163 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:45:15.242434 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239166 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:45:15.242434 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239168 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:45:15.242434 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239171 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:45:15.242434 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239173 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:45:15.242434 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239176 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:45:15.242434 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239179 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:45:15.242913 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239181 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:45:15.242913 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239184 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:45:15.242913 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239187 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:45:15.242913 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239189 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:45:15.242913 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239192 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:45:15.242913 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239195 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:45:15.242913 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239197 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:45:15.242913 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239200 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:45:15.242913 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239203 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:45:15.242913 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239207 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:45:15.242913 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239211 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:45:15.242913 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239215 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:45:15.242913 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239218 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:45:15.242913 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239221 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:45:15.242913 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239224 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:45:15.242913 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239227 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:45:15.242913 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239230 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:45:15.242913 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239233 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:45:15.242913 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239236 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:45:15.242913 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239239 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:45:15.243372 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239241 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:45:15.243372 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239244 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:45:15.243372 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239247 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:45:15.243372 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239249 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:45:15.243372 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239252 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:45:15.243372 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239254 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:45:15.243372 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239257 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:45:15.243372 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239259 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:45:15.243372 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239262 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:45:15.243372 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239264 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:45:15.243372 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239267 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:45:15.243372 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239269 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:45:15.243372 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239272 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:45:15.243372 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239274 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:45:15.243372 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239277 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:45:15.243372 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239279 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:45:15.243372 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239282 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:45:15.243372 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239285 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:45:15.243372 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239290 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:45:15.243846 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239292 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:45:15.243846 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239295 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:45:15.243846 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239297 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:45:15.243846 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239300 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:45:15.243846 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239302 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:45:15.243846 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239305 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:45:15.243846 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239308 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:45:15.243846 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239310 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:45:15.243846 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239313 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:45:15.243846 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239316 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:45:15.243846 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239318 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:45:15.243846 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239322 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:45:15.243846 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239325 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:45:15.243846 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239327 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:45:15.243846 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239330 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:45:15.243846 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239333 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:45:15.243846 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239335 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:45:15.243846 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239338 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:45:15.243846 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239340 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:45:15.243846 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239343 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:45:15.244315 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239345 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:45:15.244315 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239348 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:45:15.244315 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239350 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:45:15.244315 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239353 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:45:15.244315 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239356 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:45:15.244315 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239358 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:45:15.244315 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239360 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:45:15.244315 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239363 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:45:15.244315 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239365 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:45:15.244315 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239368 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:45:15.244315 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239371 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:45:15.244315 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239375 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:45:15.244315 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239377 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:45:15.244315 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239381 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:45:15.244315 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239384 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:45:15.244315 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239387 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:45:15.244315 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239390 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:45:15.244315 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239393 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:45:15.244315 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239396 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:45:15.244766 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239398 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:45:15.244766 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239401 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:45:15.244766 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239403 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:45:15.244766 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239407 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:45:15.244766 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239409 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:45:15.244766 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239412 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:45:15.244766 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.239414 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:45:15.244766 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239531 2578 flags.go:64] FLAG: --address="0.0.0.0" Nov 04 04:45:15.244766 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239542 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 04 04:45:15.244766 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239549 2578 flags.go:64] FLAG: --anonymous-auth="true" Nov 04 04:45:15.244766 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239554 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 04 04:45:15.244766 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239558 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 04 04:45:15.244766 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239561 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 04 04:45:15.244766 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239565 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 04 04:45:15.244766 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239570 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 04 04:45:15.244766 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239573 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 04 04:45:15.244766 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239576 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 04 04:45:15.244766 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239580 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 04 04:45:15.244766 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239583 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 04 04:45:15.244766 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239586 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 04 04:45:15.244766 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239589 2578 flags.go:64] FLAG: --cgroup-root="" Nov 04 04:45:15.244766 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239592 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 04 04:45:15.245269 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239595 2578 flags.go:64] FLAG: --client-ca-file="" Nov 04 04:45:15.245269 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239598 2578 flags.go:64] FLAG: --cloud-provider="external" Nov 04 04:45:15.245269 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239603 2578 flags.go:64] FLAG: --cluster-dns="[]" Nov 04 04:45:15.245269 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239606 2578 flags.go:64] FLAG: --cluster-domain="" Nov 04 04:45:15.245269 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239609 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 04 04:45:15.245269 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239612 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 04 04:45:15.245269 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239616 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 04 04:45:15.245269 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239619 2578 flags.go:64] FLAG: --container-log-max-files="5" Nov 04 04:45:15.245269 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239623 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 04 04:45:15.245269 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239626 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 04 04:45:15.245269 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239629 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 04 04:45:15.245269 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239633 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 04 04:45:15.245269 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239636 2578 flags.go:64] FLAG: --contention-profiling="false" Nov 04 04:45:15.245269 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239639 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 04 04:45:15.245269 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239642 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 04 04:45:15.245269 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239645 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 04 04:45:15.245269 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239648 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 04 04:45:15.245269 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239653 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 04 04:45:15.245269 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239656 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 04 04:45:15.245269 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239659 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 04 04:45:15.245269 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239662 2578 flags.go:64] FLAG: --enable-load-reader="false" Nov 04 04:45:15.245269 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239666 2578 flags.go:64] FLAG: --enable-server="true" Nov 04 04:45:15.245269 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239669 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 04 04:45:15.245269 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239673 2578 flags.go:64] FLAG: --event-burst="100" Nov 04 04:45:15.245269 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239676 2578 flags.go:64] FLAG: --event-qps="50" Nov 04 04:45:15.246380 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239679 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 04 04:45:15.246380 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239683 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 04 04:45:15.246380 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239685 2578 flags.go:64] FLAG: --eviction-hard="" Nov 04 04:45:15.246380 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239689 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 04 04:45:15.246380 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239693 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 04 04:45:15.246380 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239696 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 04 04:45:15.246380 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239699 2578 flags.go:64] FLAG: --eviction-soft="" Nov 04 04:45:15.246380 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239702 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 04 04:45:15.246380 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239706 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 04 04:45:15.246380 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239710 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 04 04:45:15.246380 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239714 2578 flags.go:64] FLAG: --experimental-mounter-path="" Nov 04 04:45:15.246380 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239717 2578 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 04 04:45:15.246380 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239720 2578 flags.go:64] FLAG: --fail-swap-on="true" Nov 04 04:45:15.246380 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239723 2578 flags.go:64] FLAG: --feature-gates="" Nov 04 04:45:15.246380 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239729 2578 flags.go:64] FLAG: --file-check-frequency="20s" Nov 04 04:45:15.246380 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239732 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 04 04:45:15.246380 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239735 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 04 04:45:15.246380 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239739 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 04 04:45:15.246380 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239742 2578 flags.go:64] FLAG: --healthz-port="10248" Nov 04 04:45:15.246380 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239746 2578 flags.go:64] FLAG: --help="false" Nov 04 04:45:15.246380 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239749 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:15.246380 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239752 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 04 04:45:15.246380 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239755 2578 flags.go:64] FLAG: --http-check-frequency="20s" Nov 04 04:45:15.246380 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239758 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 04 04:45:15.247348 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239761 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 04 04:45:15.247348 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239765 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 04 04:45:15.247348 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239768 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 04 04:45:15.247348 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239770 2578 flags.go:64] FLAG: --image-service-endpoint="" Nov 04 04:45:15.247348 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239773 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 04 04:45:15.247348 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239776 2578 flags.go:64] FLAG: --kube-api-burst="100" Nov 04 04:45:15.247348 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239779 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 04 04:45:15.247348 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239782 2578 flags.go:64] FLAG: --kube-api-qps="50" Nov 04 04:45:15.247348 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239785 2578 flags.go:64] FLAG: --kube-reserved="" Nov 04 04:45:15.247348 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239788 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 04 04:45:15.247348 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239790 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 04 04:45:15.247348 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239794 2578 flags.go:64] FLAG: --kubelet-cgroups="" Nov 04 04:45:15.247348 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239796 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 04 04:45:15.247348 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239799 2578 flags.go:64] FLAG: --lock-file="" Nov 04 04:45:15.247348 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239802 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 04 04:45:15.247348 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239805 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 04 04:45:15.247348 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239809 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 04 04:45:15.247348 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239817 2578 flags.go:64] FLAG: --log-json-split-stream="false" Nov 04 04:45:15.247348 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239820 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 04 04:45:15.247348 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239823 2578 flags.go:64] FLAG: --log-text-split-stream="false" Nov 04 04:45:15.247348 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239826 2578 flags.go:64] FLAG: --logging-format="text" Nov 04 04:45:15.247348 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239829 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 04 04:45:15.247348 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239832 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 04 04:45:15.247348 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239835 2578 flags.go:64] FLAG: --manifest-url="" Nov 04 04:45:15.248357 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239838 2578 flags.go:64] FLAG: --manifest-url-header="" Nov 04 04:45:15.248357 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239843 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 04 04:45:15.248357 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239846 2578 flags.go:64] FLAG: --max-open-files="1000000" Nov 04 04:45:15.248357 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239855 2578 flags.go:64] FLAG: --max-pods="110" Nov 04 04:45:15.248357 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239858 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 04 04:45:15.248357 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239861 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 04 04:45:15.248357 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239864 2578 flags.go:64] FLAG: --memory-manager-policy="None" Nov 04 04:45:15.248357 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239866 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 04 04:45:15.248357 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239869 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 04 04:45:15.248357 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239872 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 04 04:45:15.248357 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239875 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 04 04:45:15.248357 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239883 2578 flags.go:64] FLAG: --node-status-max-images="50" Nov 04 04:45:15.248357 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239885 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 04 04:45:15.248357 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239889 2578 flags.go:64] FLAG: --oom-score-adj="-999" Nov 04 04:45:15.248357 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239892 2578 flags.go:64] FLAG: --pod-cidr="" Nov 04 04:45:15.248357 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239894 2578 flags.go:64] FLAG: --pod-manifest-path="" Nov 04 04:45:15.248357 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239897 2578 flags.go:64] FLAG: --pod-max-pids="-1" Nov 04 04:45:15.248357 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239900 2578 flags.go:64] FLAG: --pods-per-core="0" Nov 04 04:45:15.248357 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239903 2578 flags.go:64] FLAG: --port="10250" Nov 04 04:45:15.248357 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239906 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 04 04:45:15.248357 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239909 2578 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0c85fe216f8cd8da7" Nov 04 04:45:15.248357 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239913 2578 flags.go:64] FLAG: --qos-reserved="" Nov 04 04:45:15.248357 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239915 2578 flags.go:64] FLAG: --read-only-port="10255" Nov 04 04:45:15.248357 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239919 2578 flags.go:64] FLAG: --register-node="true" Nov 04 04:45:15.248357 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239925 2578 flags.go:64] FLAG: --register-with-taints="" Nov 04 04:45:15.248357 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239930 2578 flags.go:64] FLAG: --registry-burst="10" Nov 04 04:45:15.249464 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239933 2578 flags.go:64] FLAG: --registry-qps="5" Nov 04 04:45:15.249464 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239936 2578 flags.go:64] FLAG: --reserved-cpus="" Nov 04 04:45:15.249464 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239939 2578 flags.go:64] FLAG: --reserved-memory="" Nov 04 04:45:15.249464 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239943 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 04 04:45:15.249464 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239946 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 04 04:45:15.249464 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239949 2578 flags.go:64] FLAG: --rotate-certificates="false" Nov 04 04:45:15.249464 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239952 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 04 04:45:15.249464 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239955 2578 flags.go:64] FLAG: --runonce="false" Nov 04 04:45:15.249464 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239958 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 04 04:45:15.249464 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239961 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 04 04:45:15.249464 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239965 2578 flags.go:64] FLAG: --seccomp-default="false" Nov 04 04:45:15.249464 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239968 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 04 04:45:15.249464 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239970 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 04 04:45:15.249464 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239974 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 04 04:45:15.249464 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239977 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 04 04:45:15.249464 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239980 2578 flags.go:64] FLAG: --storage-driver-password="root" Nov 04 04:45:15.249464 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239983 2578 flags.go:64] FLAG: --storage-driver-secure="false" Nov 04 04:45:15.249464 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239985 2578 flags.go:64] FLAG: --storage-driver-table="stats" Nov 04 04:45:15.249464 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239988 2578 flags.go:64] FLAG: --storage-driver-user="root" Nov 04 04:45:15.249464 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239991 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 04 04:45:15.249464 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239994 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 04 04:45:15.249464 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.239997 2578 flags.go:64] FLAG: --system-cgroups="" Nov 04 04:45:15.249464 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.240000 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 04 04:45:15.249464 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.240006 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 04 04:45:15.249464 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.240009 2578 flags.go:64] FLAG: --tls-cert-file="" Nov 04 04:45:15.250684 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.240011 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 04 04:45:15.250684 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.240015 2578 flags.go:64] FLAG: --tls-min-version="" Nov 04 04:45:15.250684 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.240017 2578 flags.go:64] FLAG: --tls-private-key-file="" Nov 04 04:45:15.250684 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.240020 2578 flags.go:64] FLAG: --topology-manager-policy="none" Nov 04 04:45:15.250684 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.240023 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 04 04:45:15.250684 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.240027 2578 flags.go:64] FLAG: --topology-manager-scope="container" Nov 04 04:45:15.250684 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.240031 2578 flags.go:64] FLAG: --v="2" Nov 04 04:45:15.250684 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.240035 2578 flags.go:64] FLAG: --version="false" Nov 04 04:45:15.250684 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.240039 2578 flags.go:64] FLAG: --vmodule="" Nov 04 04:45:15.250684 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.240043 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 04 04:45:15.250684 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.240046 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 04 04:45:15.250684 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240219 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:45:15.250684 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240224 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:45:15.250684 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240227 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:45:15.250684 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240230 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:45:15.250684 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240233 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:45:15.250684 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240236 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:45:15.250684 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240238 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:45:15.250684 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240242 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:45:15.250684 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240244 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:45:15.250684 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240247 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:45:15.250684 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240250 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:45:15.251517 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240252 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:45:15.251517 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240255 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:45:15.251517 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240257 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:45:15.251517 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240260 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:45:15.251517 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240263 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:45:15.251517 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240266 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:45:15.251517 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240268 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:45:15.251517 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240271 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:45:15.251517 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240274 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:45:15.251517 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240276 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:45:15.251517 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240279 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:45:15.251517 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240282 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:45:15.251517 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240284 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:45:15.251517 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240287 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:45:15.251517 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240290 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:45:15.251517 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240294 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:45:15.251517 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240300 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:45:15.251517 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240303 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:45:15.251517 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240307 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:45:15.252013 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240310 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:45:15.252013 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240313 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:45:15.252013 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240315 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:45:15.252013 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240318 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:45:15.252013 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240321 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:45:15.252013 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240323 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:45:15.252013 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240326 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:45:15.252013 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240328 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:45:15.252013 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240331 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:45:15.252013 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240334 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:45:15.252013 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240341 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:45:15.252013 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240345 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:45:15.252013 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240348 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:45:15.252013 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240351 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:45:15.252013 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240354 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:45:15.252013 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240357 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:45:15.252013 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240359 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:45:15.252013 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240362 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:45:15.252013 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240364 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:45:15.252827 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240367 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:45:15.252827 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240369 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:45:15.252827 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240372 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:45:15.252827 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240374 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:45:15.252827 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240377 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:45:15.252827 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240379 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:45:15.252827 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240382 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:45:15.252827 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240385 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:45:15.252827 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240387 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:45:15.252827 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240392 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:45:15.252827 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240395 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:45:15.252827 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240398 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:45:15.252827 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240401 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:45:15.252827 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240404 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:45:15.252827 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240406 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:45:15.252827 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240409 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:45:15.252827 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240411 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:45:15.252827 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240414 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:45:15.252827 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240416 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:45:15.253356 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240419 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:45:15.253356 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240421 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:45:15.253356 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240423 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:45:15.253356 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240426 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:45:15.253356 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240428 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:45:15.253356 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240431 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:45:15.253356 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240434 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:45:15.253356 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240436 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:45:15.253356 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240439 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:45:15.253356 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240442 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:45:15.253356 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240444 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:45:15.253356 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240447 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:45:15.253356 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240449 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:45:15.253356 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240452 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:45:15.253356 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240455 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:45:15.253356 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240457 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:45:15.253356 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240460 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:45:15.253356 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240462 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:45:15.253356 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240465 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:45:15.253356 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240467 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:45:15.253871 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240470 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:45:15.253871 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240472 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:45:15.253871 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240477 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:45:15.253871 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240481 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:45:15.253871 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240484 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:45:15.253871 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240487 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:45:15.253871 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240501 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:45:15.253871 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240504 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:45:15.253871 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240507 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:45:15.253871 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240510 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:45:15.253871 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240512 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:45:15.253871 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240515 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:45:15.253871 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240518 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:45:15.253871 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240520 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:45:15.253871 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240523 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:45:15.253871 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240525 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:45:15.253871 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240528 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:45:15.253871 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240531 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:45:15.254653 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.240558 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 04:45:15.254653 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240600 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:45:15.254653 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240604 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:45:15.254653 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240607 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:45:15.254653 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240610 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:45:15.254653 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240613 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:45:15.254653 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240616 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:45:15.254653 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240620 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:45:15.254653 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240624 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:45:15.254653 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240628 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:45:15.254653 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240630 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:45:15.254653 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240633 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:45:15.254653 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240636 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:45:15.254653 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240639 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:45:15.254653 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240642 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:45:15.255066 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240644 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:45:15.255066 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240647 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:45:15.255066 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240652 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:45:15.255066 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240654 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:45:15.255066 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240657 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:45:15.255066 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240660 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:45:15.255066 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240662 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:45:15.255066 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240665 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:45:15.255066 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240668 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:45:15.255066 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240670 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:45:15.255066 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240673 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:45:15.255066 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240675 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:45:15.255066 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240678 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:45:15.255066 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240681 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:45:15.255066 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240683 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:45:15.255066 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240686 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:45:15.255066 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240688 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:45:15.255066 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240691 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:45:15.255066 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240694 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:45:15.255614 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240696 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:45:15.255614 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240699 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:45:15.255614 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240701 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:45:15.255614 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240704 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:45:15.255614 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240706 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:45:15.255614 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240709 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:45:15.255614 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240711 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:45:15.255614 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240714 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:45:15.255614 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240716 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:45:15.255614 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240719 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:45:15.255614 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240722 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:45:15.255614 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240725 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:45:15.255614 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240728 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:45:15.255614 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240730 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:45:15.255614 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240733 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:45:15.255614 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240737 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:45:15.255614 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240740 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:45:15.255614 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240742 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:45:15.255614 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240744 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:45:15.255614 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240747 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:45:15.256081 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240750 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:45:15.256081 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240752 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:45:15.256081 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240754 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:45:15.256081 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240757 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:45:15.256081 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240759 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:45:15.256081 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240762 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:45:15.256081 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240765 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:45:15.256081 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240767 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:45:15.256081 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240770 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:45:15.256081 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240772 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:45:15.256081 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240775 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:45:15.256081 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240777 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:45:15.256081 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240781 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:45:15.256081 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240784 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:45:15.256081 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240786 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:45:15.256081 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240789 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:45:15.256081 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240792 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:45:15.256081 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240794 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:45:15.256081 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240797 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:45:15.256081 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240799 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:45:15.256634 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240802 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:45:15.256634 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240804 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:45:15.256634 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240807 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:45:15.256634 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240809 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:45:15.256634 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240812 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:45:15.256634 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240814 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:45:15.256634 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240817 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:45:15.256634 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240821 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:45:15.256634 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240824 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:45:15.256634 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240826 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:45:15.256634 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240829 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:45:15.256634 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240832 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:45:15.256634 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240834 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:45:15.256634 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240837 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:45:15.256634 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240839 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:45:15.256634 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240842 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:45:15.256634 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240844 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:45:15.256634 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240847 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:45:15.256634 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240850 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:45:15.256634 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240852 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:45:15.257176 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240855 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:45:15.257176 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240857 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:45:15.257176 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240860 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:45:15.257176 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240863 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:45:15.257176 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240865 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:45:15.257176 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240868 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:45:15.257176 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240873 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:45:15.257176 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240876 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:45:15.257176 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240879 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:45:15.257176 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240881 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:45:15.257176 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240884 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:45:15.257176 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240886 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:45:15.257176 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.240889 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:45:15.257176 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.249174 2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 04 04:45:15.257176 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.249193 2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 04 04:45:15.257176 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.249216 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 04 04:45:15.257176 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.249224 2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 04 04:45:15.257176 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249316 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:45:15.257618 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249325 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:45:15.257618 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249331 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:45:15.257618 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249335 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:45:15.257618 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249340 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:45:15.257618 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249344 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:45:15.257618 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249349 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:45:15.257618 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249353 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:45:15.257618 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249358 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:45:15.257618 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249362 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:45:15.257618 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249366 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:45:15.257618 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249371 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:45:15.257618 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249375 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:45:15.257618 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249379 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:45:15.257618 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249383 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:45:15.257618 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249388 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:45:15.257618 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249392 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:45:15.257618 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249396 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:45:15.257618 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249401 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:45:15.257618 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249405 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:45:15.258071 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249410 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:45:15.258071 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249415 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:45:15.258071 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249419 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:45:15.258071 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249423 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:45:15.258071 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249427 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:45:15.258071 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249432 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:45:15.258071 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249436 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:45:15.258071 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249442 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:45:15.258071 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249447 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:45:15.258071 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249451 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:45:15.258071 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249456 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:45:15.258071 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249460 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:45:15.258071 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249463 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:45:15.258071 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249467 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:45:15.258071 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249471 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:45:15.258071 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249476 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:45:15.258071 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249480 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:45:15.258071 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249483 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:45:15.258071 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249488 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:45:15.258071 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249507 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:45:15.258902 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249512 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:45:15.258902 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249517 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:45:15.258902 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249523 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:45:15.258902 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249530 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:45:15.258902 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249535 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:45:15.258902 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249539 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:45:15.258902 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249543 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:45:15.258902 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249548 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:45:15.258902 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249552 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:45:15.258902 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249556 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:45:15.258902 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249560 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:45:15.258902 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249564 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:45:15.258902 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249569 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:45:15.258902 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249573 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:45:15.258902 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249578 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:45:15.258902 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249582 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:45:15.258902 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249587 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:45:15.258902 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249592 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:45:15.258902 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249596 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:45:15.259681 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249601 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:45:15.259681 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249605 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:45:15.259681 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249614 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:45:15.259681 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249621 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:45:15.259681 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249626 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:45:15.259681 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249630 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:45:15.259681 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249635 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:45:15.259681 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249639 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:45:15.259681 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249643 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:45:15.259681 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249647 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:45:15.259681 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249651 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:45:15.259681 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249656 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:45:15.259681 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249660 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:45:15.259681 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249665 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:45:15.259681 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249669 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:45:15.259681 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249673 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:45:15.259681 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249677 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:45:15.259681 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249681 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:45:15.259681 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249686 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:45:15.260158 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249690 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:45:15.260158 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249695 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:45:15.260158 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249700 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:45:15.260158 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249704 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:45:15.260158 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249708 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:45:15.260158 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249713 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:45:15.260158 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249717 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:45:15.260158 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249721 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:45:15.260158 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249725 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:45:15.260158 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249729 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:45:15.260158 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249734 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:45:15.260158 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249738 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:45:15.260158 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249742 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:45:15.260158 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249747 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:45:15.260158 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249751 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:45:15.260158 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249755 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:45:15.260158 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249760 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:45:15.260158 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249764 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:45:15.260158 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249768 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:45:15.260158 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249772 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:45:15.260677 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249777 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:45:15.260677 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249780 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:45:15.260677 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249784 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:45:15.260677 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249788 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:45:15.260677 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249791 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:45:15.260677 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249796 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:45:15.260677 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249799 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:45:15.260677 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.249803 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:45:15.260677 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.249845 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 04:45:15.260677 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250070 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:45:15.260677 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250081 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:45:15.260677 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250086 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:45:15.260677 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250091 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:45:15.260677 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250095 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:45:15.260677 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250100 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:45:15.260677 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250105 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:45:15.261053 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250109 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:45:15.261053 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250114 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:45:15.261053 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250118 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:45:15.261053 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250122 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:45:15.261053 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250127 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:45:15.261053 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250131 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:45:15.261053 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250135 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:45:15.261053 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250140 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:45:15.261053 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250146 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:45:15.261053 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250152 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:45:15.261053 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250157 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:45:15.261053 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250163 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:45:15.261053 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250168 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:45:15.261053 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250172 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:45:15.261053 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250177 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:45:15.261053 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250181 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:45:15.261053 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250185 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:45:15.261053 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250190 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:45:15.261053 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250194 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:45:15.261504 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250198 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:45:15.261504 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250202 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:45:15.261504 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250207 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:45:15.261504 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250211 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:45:15.261504 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250215 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:45:15.261504 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250219 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:45:15.261504 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250223 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:45:15.261504 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250228 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:45:15.261504 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250233 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:45:15.261504 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250237 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:45:15.261504 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250241 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:45:15.261504 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250246 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:45:15.261504 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250251 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:45:15.261504 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250255 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:45:15.261504 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250259 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:45:15.261504 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250263 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:45:15.261504 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250267 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:45:15.261504 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250271 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:45:15.261504 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250275 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:45:15.261968 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250279 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:45:15.261968 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250283 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:45:15.261968 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250288 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:45:15.261968 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250292 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:45:15.261968 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250296 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:45:15.261968 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250300 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:45:15.261968 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250305 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:45:15.261968 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250309 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:45:15.261968 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250313 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:45:15.261968 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250318 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:45:15.261968 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250322 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:45:15.261968 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250327 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:45:15.261968 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250331 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:45:15.261968 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250336 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:45:15.261968 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250340 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:45:15.261968 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250344 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:45:15.261968 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250348 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:45:15.261968 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250352 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:45:15.261968 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250356 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:45:15.262402 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250360 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:45:15.262402 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250364 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:45:15.262402 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250368 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:45:15.262402 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250373 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:45:15.262402 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250379 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:45:15.262402 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250386 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:45:15.262402 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250391 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:45:15.262402 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250396 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:45:15.262402 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250401 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:45:15.262402 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250405 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:45:15.262402 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250409 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:45:15.262402 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250413 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:45:15.262402 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250417 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:45:15.262402 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250422 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:45:15.262402 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250426 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:45:15.262402 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250431 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:45:15.262402 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250435 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:45:15.262402 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250439 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:45:15.262402 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250443 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:45:15.262402 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250447 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:45:15.262916 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250451 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:45:15.262916 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250456 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:45:15.262916 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250460 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:45:15.262916 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250464 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:45:15.262916 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250468 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:45:15.262916 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250473 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:45:15.262916 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250477 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:45:15.262916 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250481 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:45:15.262916 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250486 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:45:15.262916 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250506 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:45:15.262916 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250510 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:45:15.262916 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250515 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:45:15.262916 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250519 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:45:15.262916 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250522 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:45:15.262916 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250527 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:45:15.262916 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250531 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:45:15.262916 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250536 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:45:15.262916 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250540 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:45:15.262916 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250544 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:45:15.263344 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250549 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:45:15.263344 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250554 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:45:15.263344 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250558 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:45:15.263344 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.250597 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 04:45:15.263344 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250648 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:45:15.263344 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250655 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:45:15.263344 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250662 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:45:15.263344 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250666 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:45:15.263344 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250670 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:45:15.263344 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250675 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:45:15.263344 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250679 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:45:15.263344 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250683 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:45:15.263344 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250687 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:45:15.263344 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250691 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:45:15.263344 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250695 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:45:15.263712 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250699 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:45:15.263712 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250703 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:45:15.263712 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250707 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:45:15.263712 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250711 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:45:15.263712 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250715 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:45:15.263712 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250720 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:45:15.263712 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250724 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:45:15.263712 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250728 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:45:15.263712 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250733 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:45:15.263712 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250737 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:45:15.263712 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250741 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:45:15.263712 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250745 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:45:15.263712 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250749 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:45:15.263712 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250753 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:45:15.263712 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250757 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:45:15.263712 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250761 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:45:15.263712 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250766 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:45:15.263712 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250770 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:45:15.263712 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250774 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:45:15.264145 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250780 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:45:15.264145 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250784 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:45:15.264145 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250789 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:45:15.264145 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250795 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:45:15.264145 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250803 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:45:15.264145 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250809 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:45:15.264145 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250813 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:45:15.264145 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250818 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:45:15.264145 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250822 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:45:15.264145 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250826 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:45:15.264145 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250831 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:45:15.264145 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250834 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:45:15.264145 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250839 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:45:15.264145 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250843 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:45:15.264145 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250848 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:45:15.264145 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250852 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:45:15.264145 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250856 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:45:15.264145 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250860 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:45:15.264145 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250865 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:45:15.264145 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250869 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:45:15.264633 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250873 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:45:15.264633 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250878 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:45:15.264633 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250882 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:45:15.264633 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250886 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:45:15.264633 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250890 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:45:15.264633 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250894 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:45:15.264633 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250899 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:45:15.264633 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250903 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:45:15.264633 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250907 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:45:15.264633 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250911 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:45:15.264633 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250916 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:45:15.264633 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250920 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:45:15.264633 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250924 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:45:15.264633 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250930 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:45:15.264633 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250935 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:45:15.264633 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250941 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:45:15.264633 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250947 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:45:15.264633 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250951 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:45:15.264633 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250955 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:45:15.265066 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250960 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:45:15.265066 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250964 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:45:15.265066 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250968 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:45:15.265066 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250972 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:45:15.265066 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250977 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:45:15.265066 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250982 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:45:15.265066 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250986 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:45:15.265066 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250990 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:45:15.265066 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250994 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:45:15.265066 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.250999 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:45:15.265066 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.251003 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:45:15.265066 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.251007 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:45:15.265066 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.251012 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:45:15.265066 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.251016 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:45:15.265066 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.251021 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:45:15.265066 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.251025 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:45:15.265066 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.251029 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:45:15.265066 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.251035 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:45:15.265066 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.251040 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:45:15.265066 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.251045 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:45:15.265533 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.251077 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:45:15.265533 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.251082 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:45:15.265533 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.251087 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:45:15.265533 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.251091 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:45:15.265533 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.251095 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:45:15.265533 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.251099 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:45:15.265533 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.251104 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:45:15.265533 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.251109 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:45:15.265533 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.251115 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:45:15.265533 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.251120 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:45:15.265533 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.251125 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:45:15.265533 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.251129 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:45:15.265533 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.251133 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:45:15.265533 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.251137 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:45:15.265533 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.251141 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:45:15.265533 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.251146 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:45:15.265533 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:15.251151 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:45:15.265533 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.251231 2578 server.go:957] "Client rotation is on, will bootstrap in background" Nov 04 04:45:15.265533 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.255039 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 04 04:45:15.265975 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.256124 2578 server.go:1014] "Starting client certificate rotation" Nov 04 04:45:15.265975 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.256242 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 04:45:15.265975 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.256276 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 04:45:15.265975 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.258733 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 04:45:15.265975 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.260254 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 04:45:15.272954 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.272939 2578 log.go:25] "Validated CRI v1 runtime API" Nov 04 04:45:15.277183 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.276955 2578 log.go:25] "Validated CRI v1 image API" Nov 04 04:45:15.278379 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.278354 2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 04 04:45:15.283261 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.283241 2578 fs.go:135] Filesystem UUIDs: map[0a8d1a56-04db-40ac-965d-9ac5017686fe:/dev/nvme0n1p4 74c82a04-1aa8-43b9-a324-498152680015:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Nov 04 04:45:15.283348 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.283260 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 04 04:45:15.291643 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.291529 2578 manager.go:217] Machine: {Timestamp:2026-11-04 04:45:15.290126115 +0000 UTC m=+0.384499964 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098360 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c026f93b8c85f8b97ef083c4a8115 SystemUUID:ec2c026f-93b8-c85f-8b97-ef083c4a8115 BootID:e21ebccf-c7d6-4baf-9c55-6579bf3f2121 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:f3:20:b0:db Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:f3:20:b0:db Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b6:36:f1:f6:c7:b9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 04 04:45:15.292218 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.292200 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 04 04:45:15.292292 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.292280 2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 04 04:45:15.293530 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.293506 2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 04 04:45:15.293680 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.293532 2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-224.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 04 04:45:15.293771 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.293684 2578 topology_manager.go:143] "Creating topology manager with none policy" Nov 04 04:45:15.293771 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.293695 2578 container_manager_linux.go:311] "Creating device plugin manager" Nov 04 04:45:15.293771 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.293740 2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 04:45:15.293771 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.293767 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 04:45:15.294759 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.294746 2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 04 04:45:15.298104 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.298089 2578 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 04 04:45:15.298226 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.298215 2578 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 04 04:45:15.299061 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.299050 2578 kubelet.go:493] "Attempting to sync node with API server" Nov 04 04:45:15.299116 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.299067 2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 04 04:45:15.299116 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.299081 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 04 04:45:15.299116 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.299093 2578 kubelet.go:405] "Adding apiserver pod source" Nov 04 04:45:15.299116 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.299104 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 04 04:45:15.299473 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.299459 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 04:45:15.301960 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.301940 2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 04 04:45:15.307621 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.307608 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 04:45:15.308745 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.308734 2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 04 04:45:15.308782 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.308763 2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 04 04:45:15.308909 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.308899 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 04 04:45:15.308936 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.308919 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 04 04:45:15.308936 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.308929 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 04 04:45:15.308986 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.308939 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 04 04:45:15.308986 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.308946 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 04 04:45:15.308986 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.308954 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 04 04:45:15.309667 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.309658 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 04 04:45:15.309698 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.309668 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 04 04:45:15.309698 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.309675 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 04 04:45:15.309698 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.309681 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 04 04:45:15.309698 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.309690 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 04 04:45:15.309893 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.309700 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 04 04:45:15.310283 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.310274 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 04 04:45:15.310316 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.310283 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 04 04:45:15.313914 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.313902 2578 server.go:1263] "Started kubelet" Nov 04 04:45:15.313976 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.313951 2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 04 04:45:15.314063 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.314008 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 04 04:45:15.314116 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.314103 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 04 04:45:15.314648 ip-10-0-134-224 systemd[1]: Started Kubernetes Kubelet. Nov 04 04:45:15.315547 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.315199 2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 04 04:45:15.317768 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.317756 2578 server.go:317] "Adding debug handlers to kubelet server" Nov 04 04:45:15.322393 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.322377 2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 04 04:45:15.322474 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.322412 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 04 04:45:15.322589 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:15.322572 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-224.us-west-2.compute.internal\" not found" Nov 04 04:45:15.323168 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.323151 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 04 04:45:15.323168 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.323153 2578 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 04 04:45:15.323305 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.323180 2578 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 04 04:45:15.323353 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.323310 2578 reconstruct.go:97] "Volume reconstruction finished" Nov 04 04:45:15.323353 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.323322 2578 reconciler.go:29] "Reconciler: start to sync state" Nov 04 04:45:15.323805 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:15.323778 2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 04 04:45:15.324185 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.324108 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 04 04:45:15.324185 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.324124 2578 factory.go:55] Registering systemd factory Nov 04 04:45:15.324185 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.324133 2578 factory.go:223] Registration of the systemd container factory successfully Nov 04 04:45:15.324372 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.324342 2578 factory.go:153] Registering CRI-O factory Nov 04 04:45:15.324372 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.324353 2578 factory.go:223] Registration of the crio container factory successfully Nov 04 04:45:15.324372 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.324366 2578 factory.go:103] Registering Raw factory Nov 04 04:45:15.324372 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.324375 2578 manager.go:1199] Started watching for new ooms in manager Nov 04 04:45:15.325296 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.325281 2578 manager.go:319] Starting recovery of all containers Nov 04 04:45:15.335757 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.335742 2578 manager.go:324] Recovery completed Nov 04 04:45:15.339534 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.339522 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:45:15.341461 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.341442 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:45:15.341567 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.341472 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:45:15.341567 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.341482 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 04:45:15.341972 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.341961 2578 cpu_manager.go:226] "Starting" policy="none" Nov 04 04:45:15.341972 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.341971 2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 04 04:45:15.342044 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.341972 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:45:15.342044 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.341985 2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 04 04:45:15.342640 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.342625 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:45:15.342707 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.342655 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:45:15.342707 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.342669 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 04:45:15.343699 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.343686 2578 policy_none.go:50] "Start" Nov 04 04:45:15.343699 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.343701 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 04 04:45:15.343780 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.343710 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 04 04:45:15.345089 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.345079 2578 policy_none.go:44] "Start" Nov 04 04:45:15.377038 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.377015 2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 04 04:45:15.380183 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.380171 2578 manager.go:341] "Starting Device Plugin manager" Nov 04 04:45:15.380261 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:15.380200 2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 04 04:45:15.380261 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.380215 2578 server.go:87] "Starting device plugin registration server" Nov 04 04:45:15.380444 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.380429 2578 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 04 04:45:15.380562 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.380445 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 04 04:45:15.380625 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.380606 2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 04 04:45:15.380688 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.380675 2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 04 04:45:15.380743 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.380688 2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 04 04:45:15.381196 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:15.381162 2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 04 04:45:15.381271 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:15.381229 2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-224.us-west-2.compute.internal\" not found" Nov 04 04:45:15.448884 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.448856 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 04 04:45:15.450035 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.449965 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 04 04:45:15.450035 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.449986 2578 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 04 04:45:15.450035 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.450002 2578 kubelet.go:2521] "Starting kubelet main sync loop" Nov 04 04:45:15.450035 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:15.450031 2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 04 04:45:15.454535 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:15.454511 2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-224.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 04 04:45:15.454671 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.454654 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-224.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 04 04:45:15.456715 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:15.454481 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-224.us-west-2.compute.internal.18e4be0d27eb09e1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-224.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-224.us-west-2.compute.internal,},FirstTimestamp:2026-11-04 04:45:15.313867233 +0000 UTC m=+0.408241081,LastTimestamp:2026-11-04 04:45:15.313867233 +0000 UTC m=+0.408241081,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-224.us-west-2.compute.internal,}" Nov 04 04:45:15.481367 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.481349 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:45:15.482007 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.481993 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:45:15.482078 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.482022 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:45:15.482078 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.482032 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 04:45:15.482078 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.482055 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:15.526546 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.526365 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 04 04:45:15.526546 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.526445 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zswfw" Nov 04 04:45:15.550967 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.550946 2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-224.us-west-2.compute.internal"] Nov 04 04:45:15.551031 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.550992 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:45:15.551684 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.551667 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:45:15.551750 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.551695 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:45:15.551750 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.551707 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 04:45:15.553100 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.553088 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:45:15.553245 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.553234 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:15.553281 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.553258 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:45:15.553763 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.553751 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:45:15.553829 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.553780 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:45:15.553829 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.553795 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 04:45:15.553909 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.553751 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:45:15.553909 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.553876 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:45:15.553909 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.553889 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 04:45:15.555443 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.555428 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:15.555552 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.555451 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:45:15.556095 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.556077 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:45:15.556182 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.556107 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:45:15.556182 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.556119 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 04:45:15.587315 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:15.587286 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-224.us-west-2.compute.internal\" not found" node="ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:15.591053 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:15.591036 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-224.us-west-2.compute.internal\" not found" node="ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:15.614168 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.613854 2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 04:45:15.625525 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.625487 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/95a50811676d3438ddaa64148db61400-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal\" (UID: \"95a50811676d3438ddaa64148db61400\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:15.625589 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.625554 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95a50811676d3438ddaa64148db61400-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal\" (UID: \"95a50811676d3438ddaa64148db61400\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:15.625627 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.625599 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/88d814719e03f487d48b25be7946048f-config\") pod \"kube-apiserver-proxy-ip-10-0-134-224.us-west-2.compute.internal\" (UID: \"88d814719e03f487d48b25be7946048f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:15.705094 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.705073 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 04:45:15.706240 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.706220 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 04:45:15.706342 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.706326 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 04:45:15.707637 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.707620 2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:15.707682 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:15.707646 2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-224.us-west-2.compute.internal\": node \"ip-10-0-134-224.us-west-2.compute.internal\" not found" Nov 04 04:45:15.723479 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.723457 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:15.726622 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.726606 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/95a50811676d3438ddaa64148db61400-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal\" (UID: \"95a50811676d3438ddaa64148db61400\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:15.726701 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.726628 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95a50811676d3438ddaa64148db61400-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal\" (UID: \"95a50811676d3438ddaa64148db61400\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:15.726701 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.726644 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/88d814719e03f487d48b25be7946048f-config\") pod \"kube-apiserver-proxy-ip-10-0-134-224.us-west-2.compute.internal\" (UID: \"88d814719e03f487d48b25be7946048f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:15.726769 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.726700 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/88d814719e03f487d48b25be7946048f-config\") pod \"kube-apiserver-proxy-ip-10-0-134-224.us-west-2.compute.internal\" (UID: \"88d814719e03f487d48b25be7946048f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:15.726769 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.726714 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/95a50811676d3438ddaa64148db61400-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal\" (UID: \"95a50811676d3438ddaa64148db61400\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:15.726769 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.726724 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95a50811676d3438ddaa64148db61400-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal\" (UID: \"95a50811676d3438ddaa64148db61400\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:15.786679 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.786635 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 04:45:15.786751 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.786716 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:15.842213 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.842192 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 04:45:15.890572 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.890550 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:15.895595 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:15.895185 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:16.255393 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:16.255339 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 04 04:45:16.255911 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:16.255486 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 04 04:45:16.255911 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:16.255510 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 04 04:45:16.255911 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:16.255549 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 04 04:45:16.255911 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:16.255482 2578 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://afa22c3b1a41d46a3a14deb84ca41375-9501ff5f8f77d258.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-134-224.us-west-2.compute.internal.18e4be0d29900a7e\": read tcp 10.0.134.224:44726->3.22.197.16:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-134-224.us-west-2.compute.internal.18e4be0d29900a7e default 5201 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-224.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-224.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-224.us-west-2.compute.internal,},FirstTimestamp:2026-11-04 04:45:15 +0000 UTC,LastTimestamp:2026-11-04 04:45:15.551683511 +0000 UTC m=+0.646057362,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-224.us-west-2.compute.internal,}" Nov 04 04:45:17.395083 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.395059 2578 apiserver.go:52] "Watching apiserver" Nov 04 04:45:17.419218 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.419193 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 04 04:45:17.626227 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.626163 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-03 04:40:16 +0000 UTC" deadline="2028-07-14 23:29:01.562844732 +0000 UTC" Nov 04 04:45:17.626227 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.626197 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14850h43m43.936651656s" Nov 04 04:45:17.710309 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.710271 2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 04 04:45:17.710612 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.710593 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-dj7mc","openshift-image-registry/node-ca-7tlqm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-p9jxj","openshift-multus/multus-m692r","openshift-multus/network-metrics-daemon-tkgd9","kube-system/global-pull-secret-syncer-mwgm5","kube-system/kube-apiserver-proxy-ip-10-0-134-224.us-west-2.compute.internal","openshift-network-diagnostics/network-check-target-mfn6w","openshift-dns/node-resolver-7f22t","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx","openshift-network-operator/iptables-alerter-q6grq","openshift-ovn-kubernetes/ovnkube-node-dzb65","kube-system/konnectivity-agent-ct2mt"] Nov 04 04:45:17.712166 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.712146 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.713488 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.713465 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7tlqm" Nov 04 04:45:17.714925 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.714902 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:17.716226 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.716207 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-m692r" Nov 04 04:45:17.717576 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.717557 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkgd9" Nov 04 04:45:17.717657 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:17.717624 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkgd9" podUID="f0c260a1-2a73-4b2b-b6a7-2f9717941a1b" Nov 04 04:45:17.718778 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.718760 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:17.719894 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.719877 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfn6w" Nov 04 04:45:17.719989 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:17.719926 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfn6w" podUID="59d18119-72a5-410f-be97-90613af651bd" Nov 04 04:45:17.720111 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:17.720088 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwgm5" podUID="3e3bf36f-7156-4151-9d6d-0937169dbcca" Nov 04 04:45:17.721199 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.721178 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7f22t" Nov 04 04:45:17.723691 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.723668 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:17.724527 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.724509 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-q6grq" Nov 04 04:45:17.725965 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.725946 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.726242 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.726220 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-cni-binary-copy\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.726369 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.726275 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-host-run-k8s-cni-cncf-io\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.726369 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.726362 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-host-var-lib-kubelet\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.726477 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.726384 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-etc-kubernetes\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.726477 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.726399 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3e3bf36f-7156-4151-9d6d-0937169dbcca-kubelet-config\") pod \"global-pull-secret-syncer-mwgm5\" (UID: \"3e3bf36f-7156-4151-9d6d-0937169dbcca\") " pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:17.726477 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.726420 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-modprobe-d\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.726477 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.726443 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-sysctl-d\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.726477 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.726460 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-run\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.726477 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.726474 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-multus-cni-dir\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.726742 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.726489 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/31e07d9c-93bc-4e8d-909b-e65406ff62df-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p9jxj\" (UID: \"31e07d9c-93bc-4e8d-909b-e65406ff62df\") " pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:17.726742 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.726513 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd5qw\" (UniqueName: \"kubernetes.io/projected/f0c260a1-2a73-4b2b-b6a7-2f9717941a1b-kube-api-access-pd5qw\") pod \"network-metrics-daemon-tkgd9\" (UID: \"f0c260a1-2a73-4b2b-b6a7-2f9717941a1b\") " pod="openshift-multus/network-metrics-daemon-tkgd9" Nov 04 04:45:17.726742 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.726536 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-systemd\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.726742 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.726560 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp26f\" (UniqueName: \"kubernetes.io/projected/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-kube-api-access-kp26f\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.726742 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.726580 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-hostroot\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.726742 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.726615 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3e3bf36f-7156-4151-9d6d-0937169dbcca-original-pull-secret\") pod \"global-pull-secret-syncer-mwgm5\" (UID: \"3e3bf36f-7156-4151-9d6d-0937169dbcca\") " pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:17.726742 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.726643 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31e07d9c-93bc-4e8d-909b-e65406ff62df-system-cni-dir\") pod \"multus-additional-cni-plugins-p9jxj\" (UID: \"31e07d9c-93bc-4e8d-909b-e65406ff62df\") " pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:17.726742 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.726687 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-kubernetes\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.726742 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.726711 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-host\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.727143 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.726749 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-tuned\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.727143 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.726771 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c1153d6-83db-4eb7-95e0-78d09bd61d6d-host\") pod \"node-ca-7tlqm\" (UID: \"5c1153d6-83db-4eb7-95e0-78d09bd61d6d\") " pod="openshift-image-registry/node-ca-7tlqm" Nov 04 04:45:17.727143 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.726800 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-host-var-lib-cni-bin\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.727143 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.726815 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-multus-daemon-config\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.727143 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.726837 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6kb6\" (UniqueName: \"kubernetes.io/projected/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-kube-api-access-w6kb6\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.727143 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.726853 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-host-run-netns\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.727143 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.726873 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-sysctl-conf\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.727143 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.726898 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-sys\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.727143 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.726916 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-lib-modules\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.727143 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.726929 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-var-lib-kubelet\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.727143 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.726942 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-tmp\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.727143 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.726965 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3e3bf36f-7156-4151-9d6d-0937169dbcca-dbus\") pod \"global-pull-secret-syncer-mwgm5\" (UID: \"3e3bf36f-7156-4151-9d6d-0937169dbcca\") " pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:17.727143 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.726981 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/31e07d9c-93bc-4e8d-909b-e65406ff62df-cnibin\") pod \"multus-additional-cni-plugins-p9jxj\" (UID: \"31e07d9c-93bc-4e8d-909b-e65406ff62df\") " pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:17.727143 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.727000 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-sysconfig\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.727143 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.727015 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-system-cni-dir\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.727143 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.727037 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/31e07d9c-93bc-4e8d-909b-e65406ff62df-cni-binary-copy\") pod \"multus-additional-cni-plugins-p9jxj\" (UID: \"31e07d9c-93bc-4e8d-909b-e65406ff62df\") " pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:17.727143 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.727063 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/31e07d9c-93bc-4e8d-909b-e65406ff62df-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p9jxj\" (UID: \"31e07d9c-93bc-4e8d-909b-e65406ff62df\") " pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:17.727946 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.727082 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lppk7\" (UniqueName: \"kubernetes.io/projected/31e07d9c-93bc-4e8d-909b-e65406ff62df-kube-api-access-lppk7\") pod \"multus-additional-cni-plugins-p9jxj\" (UID: \"31e07d9c-93bc-4e8d-909b-e65406ff62df\") " pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:17.727946 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.727096 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0c260a1-2a73-4b2b-b6a7-2f9717941a1b-metrics-certs\") pod \"network-metrics-daemon-tkgd9\" (UID: \"f0c260a1-2a73-4b2b-b6a7-2f9717941a1b\") " pod="openshift-multus/network-metrics-daemon-tkgd9" Nov 04 04:45:17.727946 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.727111 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7jdp\" (UniqueName: \"kubernetes.io/projected/59d18119-72a5-410f-be97-90613af651bd-kube-api-access-b7jdp\") pod \"network-check-target-mfn6w\" (UID: \"59d18119-72a5-410f-be97-90613af651bd\") " pod="openshift-network-diagnostics/network-check-target-mfn6w" Nov 04 04:45:17.727946 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.727131 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-cnibin\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.727946 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.727154 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-multus-socket-dir-parent\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.727946 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.727170 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-host-var-lib-cni-multus\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.727946 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.727184 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5c1153d6-83db-4eb7-95e0-78d09bd61d6d-serviceca\") pod \"node-ca-7tlqm\" (UID: \"5c1153d6-83db-4eb7-95e0-78d09bd61d6d\") " pod="openshift-image-registry/node-ca-7tlqm" Nov 04 04:45:17.727946 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.727214 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d9jw\" (UniqueName: \"kubernetes.io/projected/5c1153d6-83db-4eb7-95e0-78d09bd61d6d-kube-api-access-5d9jw\") pod \"node-ca-7tlqm\" (UID: \"5c1153d6-83db-4eb7-95e0-78d09bd61d6d\") " pod="openshift-image-registry/node-ca-7tlqm" Nov 04 04:45:17.727946 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.727259 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-multus-conf-dir\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.727946 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.727278 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-host-run-multus-certs\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.727946 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.727293 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/31e07d9c-93bc-4e8d-909b-e65406ff62df-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p9jxj\" (UID: \"31e07d9c-93bc-4e8d-909b-e65406ff62df\") " pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:17.727946 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.727357 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ct2mt" Nov 04 04:45:17.772118 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.771991 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 04 04:45:17.772118 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.771995 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-l2xwd\"" Nov 04 04:45:17.772241 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.772168 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 04 04:45:17.773265 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.773249 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 04 04:45:17.773265 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.773248 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-nhf59\"" Nov 04 04:45:17.773418 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.773347 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 04 04:45:17.773418 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.773381 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 04 04:45:17.774623 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.774607 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 04 04:45:17.774813 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.774798 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 04 04:45:17.774962 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.774951 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 04 04:45:17.775070 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.775056 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 04 04:45:17.775112 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.775089 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ss4dp\"" Nov 04 04:45:17.775167 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.775155 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 04 04:45:17.776161 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.776148 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 04 04:45:17.776224 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.776213 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-gwbb5\"" Nov 04 04:45:17.781406 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.781387 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 04 04:45:17.781586 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.781571 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7qqph\"" Nov 04 04:45:17.781682 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.781668 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 04 04:45:17.784098 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.784075 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 04 04:45:17.784098 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.784085 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 04 04:45:17.784269 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.784183 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 04 04:45:17.784269 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.784206 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8xbd4\"" Nov 04 04:45:17.819797 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.819783 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 04 04:45:17.827568 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.827550 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-kubernetes\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.827641 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.827574 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-host\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.827641 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.827589 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c1153d6-83db-4eb7-95e0-78d09bd61d6d-host\") pod \"node-ca-7tlqm\" (UID: \"5c1153d6-83db-4eb7-95e0-78d09bd61d6d\") " pod="openshift-image-registry/node-ca-7tlqm" Nov 04 04:45:17.827641 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.827637 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-multus-daemon-config\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.827741 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.827652 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-host\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.827741 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.827662 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-node-log\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.827741 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.827680 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c1153d6-83db-4eb7-95e0-78d09bd61d6d-host\") pod \"node-ca-7tlqm\" (UID: \"5c1153d6-83db-4eb7-95e0-78d09bd61d6d\") " pod="openshift-image-registry/node-ca-7tlqm" Nov 04 04:45:17.827741 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.827685 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-kubernetes\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.827741 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.827686 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7380461b-403d-4470-8c1c-6433616b1dd2-ovnkube-config\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.827741 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.827719 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a7936867-0a26-4c2e-b94d-a1756cd14873-agent-certs\") pod \"konnectivity-agent-ct2mt\" (UID: \"a7936867-0a26-4c2e-b94d-a1756cd14873\") " pod="kube-system/konnectivity-agent-ct2mt" Nov 04 04:45:17.827982 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.827743 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0c260a1-2a73-4b2b-b6a7-2f9717941a1b-metrics-certs\") pod \"network-metrics-daemon-tkgd9\" (UID: \"f0c260a1-2a73-4b2b-b6a7-2f9717941a1b\") " pod="openshift-multus/network-metrics-daemon-tkgd9" Nov 04 04:45:17.827982 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.827808 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gchdh\" (UniqueName: \"kubernetes.io/projected/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-kube-api-access-gchdh\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:17.827982 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.827834 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-host-run-netns\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.827982 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:17.827906 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 04:45:17.827982 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.827915 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-host-run-netns\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.828225 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:17.827987 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0c260a1-2a73-4b2b-b6a7-2f9717941a1b-metrics-certs podName:f0c260a1-2a73-4b2b-b6a7-2f9717941a1b nodeName:}" failed. No retries permitted until 2026-11-04 04:45:18.32795471 +0000 UTC m=+2.326523579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f0c260a1-2a73-4b2b-b6a7-2f9717941a1b-metrics-certs") pod "network-metrics-daemon-tkgd9" (UID: "f0c260a1-2a73-4b2b-b6a7-2f9717941a1b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 04:45:17.828225 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828017 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-sysctl-conf\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.828225 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828047 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-var-lib-kubelet\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.828225 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828072 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-tmp\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.828225 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828076 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-multus-daemon-config\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.828225 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828096 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3e3bf36f-7156-4151-9d6d-0937169dbcca-dbus\") pod \"global-pull-secret-syncer-mwgm5\" (UID: \"3e3bf36f-7156-4151-9d6d-0937169dbcca\") " pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:17.828225 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828122 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-var-lib-kubelet\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.828225 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828121 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/31e07d9c-93bc-4e8d-909b-e65406ff62df-cnibin\") pod \"multus-additional-cni-plugins-p9jxj\" (UID: \"31e07d9c-93bc-4e8d-909b-e65406ff62df\") " pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:17.828225 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828147 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/31e07d9c-93bc-4e8d-909b-e65406ff62df-cni-binary-copy\") pod \"multus-additional-cni-plugins-p9jxj\" (UID: \"31e07d9c-93bc-4e8d-909b-e65406ff62df\") " pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:17.828225 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828162 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lppk7\" (UniqueName: \"kubernetes.io/projected/31e07d9c-93bc-4e8d-909b-e65406ff62df-kube-api-access-lppk7\") pod \"multus-additional-cni-plugins-p9jxj\" (UID: \"31e07d9c-93bc-4e8d-909b-e65406ff62df\") " pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:17.828225 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828167 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/31e07d9c-93bc-4e8d-909b-e65406ff62df-cnibin\") pod \"multus-additional-cni-plugins-p9jxj\" (UID: \"31e07d9c-93bc-4e8d-909b-e65406ff62df\") " pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:17.828225 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828098 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-sysctl-conf\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.828225 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828185 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7jdp\" (UniqueName: \"kubernetes.io/projected/59d18119-72a5-410f-be97-90613af651bd-kube-api-access-b7jdp\") pod \"network-check-target-mfn6w\" (UID: \"59d18119-72a5-410f-be97-90613af651bd\") " pod="openshift-network-diagnostics/network-check-target-mfn6w" Nov 04 04:45:17.828225 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828186 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3e3bf36f-7156-4151-9d6d-0937169dbcca-dbus\") pod \"global-pull-secret-syncer-mwgm5\" (UID: \"3e3bf36f-7156-4151-9d6d-0937169dbcca\") " pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:17.828225 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828206 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-device-dir\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:17.828225 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828221 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-var-lib-openvswitch\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.828862 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828235 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-run-ovn\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.828862 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828271 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwmfd\" (UniqueName: \"kubernetes.io/projected/7380461b-403d-4470-8c1c-6433616b1dd2-kube-api-access-pwmfd\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.828862 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828308 2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 04 04:45:17.828862 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828322 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cab63986-8b41-42c4-b7b0-4438e54fa424-iptables-alerter-script\") pod \"iptables-alerter-q6grq\" (UID: \"cab63986-8b41-42c4-b7b0-4438e54fa424\") " pod="openshift-network-operator/iptables-alerter-q6grq" Nov 04 04:45:17.828862 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828381 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-etc-selinux\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:17.828862 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828410 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-run-netns\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.828862 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828435 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a7936867-0a26-4c2e-b94d-a1756cd14873-konnectivity-ca\") pod \"konnectivity-agent-ct2mt\" (UID: \"a7936867-0a26-4c2e-b94d-a1756cd14873\") " pod="kube-system/konnectivity-agent-ct2mt" Nov 04 04:45:17.828862 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828451 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5c1153d6-83db-4eb7-95e0-78d09bd61d6d-serviceca\") pod \"node-ca-7tlqm\" (UID: \"5c1153d6-83db-4eb7-95e0-78d09bd61d6d\") " pod="openshift-image-registry/node-ca-7tlqm" Nov 04 04:45:17.828862 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828466 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-multus-conf-dir\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.828862 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828503 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-host-run-multus-certs\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.828862 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828533 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-host-run-multus-certs\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.828862 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828550 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/31e07d9c-93bc-4e8d-909b-e65406ff62df-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p9jxj\" (UID: \"31e07d9c-93bc-4e8d-909b-e65406ff62df\") " pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:17.828862 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828582 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-systemd-units\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.828862 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828606 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zszmj\" (UniqueName: \"kubernetes.io/projected/cab63986-8b41-42c4-b7b0-4438e54fa424-kube-api-access-zszmj\") pod \"iptables-alerter-q6grq\" (UID: \"cab63986-8b41-42c4-b7b0-4438e54fa424\") " pod="openshift-network-operator/iptables-alerter-q6grq" Nov 04 04:45:17.828862 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828631 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-etc-kubernetes\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.828862 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828642 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-multus-conf-dir\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.828862 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828655 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3e3bf36f-7156-4151-9d6d-0937169dbcca-kubelet-config\") pod \"global-pull-secret-syncer-mwgm5\" (UID: \"3e3bf36f-7156-4151-9d6d-0937169dbcca\") " pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:17.829502 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828660 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/31e07d9c-93bc-4e8d-909b-e65406ff62df-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p9jxj\" (UID: \"31e07d9c-93bc-4e8d-909b-e65406ff62df\") " pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:17.829502 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828687 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7380461b-403d-4470-8c1c-6433616b1dd2-ovn-node-metrics-cert\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.829502 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828713 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7380461b-403d-4470-8c1c-6433616b1dd2-ovnkube-script-lib\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.829502 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828717 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3e3bf36f-7156-4151-9d6d-0937169dbcca-kubelet-config\") pod \"global-pull-secret-syncer-mwgm5\" (UID: \"3e3bf36f-7156-4151-9d6d-0937169dbcca\") " pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:17.829502 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828746 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-etc-kubernetes\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.829502 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828748 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd5qw\" (UniqueName: \"kubernetes.io/projected/f0c260a1-2a73-4b2b-b6a7-2f9717941a1b-kube-api-access-pd5qw\") pod \"network-metrics-daemon-tkgd9\" (UID: \"f0c260a1-2a73-4b2b-b6a7-2f9717941a1b\") " pod="openshift-multus/network-metrics-daemon-tkgd9" Nov 04 04:45:17.829502 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828745 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/31e07d9c-93bc-4e8d-909b-e65406ff62df-cni-binary-copy\") pod \"multus-additional-cni-plugins-p9jxj\" (UID: \"31e07d9c-93bc-4e8d-909b-e65406ff62df\") " pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:17.829502 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828781 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:17.829502 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828831 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-socket-dir\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:17.829502 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828860 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-systemd\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.829502 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828899 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-systemd\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.829502 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828900 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5c1153d6-83db-4eb7-95e0-78d09bd61d6d-serviceca\") pod \"node-ca-7tlqm\" (UID: \"5c1153d6-83db-4eb7-95e0-78d09bd61d6d\") " pod="openshift-image-registry/node-ca-7tlqm" Nov 04 04:45:17.829502 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828917 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-hostroot\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.829502 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.828958 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3e3bf36f-7156-4151-9d6d-0937169dbcca-original-pull-secret\") pod \"global-pull-secret-syncer-mwgm5\" (UID: \"3e3bf36f-7156-4151-9d6d-0937169dbcca\") " pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:17.829502 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829002 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31e07d9c-93bc-4e8d-909b-e65406ff62df-system-cni-dir\") pod \"multus-additional-cni-plugins-p9jxj\" (UID: \"31e07d9c-93bc-4e8d-909b-e65406ff62df\") " pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:17.829502 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829009 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-hostroot\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.829502 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:17.829043 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 04:45:17.830208 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829075 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-sys-fs\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:17.830208 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829090 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31e07d9c-93bc-4e8d-909b-e65406ff62df-system-cni-dir\") pod \"multus-additional-cni-plugins-p9jxj\" (UID: \"31e07d9c-93bc-4e8d-909b-e65406ff62df\") " pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:17.830208 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829118 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-kubelet\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.830208 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829133 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-tuned\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.830208 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829168 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-host-var-lib-cni-bin\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.830208 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:17.829195 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e3bf36f-7156-4151-9d6d-0937169dbcca-original-pull-secret podName:3e3bf36f-7156-4151-9d6d-0937169dbcca nodeName:}" failed. No retries permitted until 2026-11-04 04:45:18.329182263 +0000 UTC m=+2.327751130 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3e3bf36f-7156-4151-9d6d-0937169dbcca-original-pull-secret") pod "global-pull-secret-syncer-mwgm5" (UID: "3e3bf36f-7156-4151-9d6d-0937169dbcca") : object "kube-system"/"original-pull-secret" not registered Nov 04 04:45:17.830208 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829248 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6kb6\" (UniqueName: \"kubernetes.io/projected/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-kube-api-access-w6kb6\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.830208 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829262 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-host-var-lib-cni-bin\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.830208 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829289 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-slash\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.830208 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829330 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-run-systemd\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.830208 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829357 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-log-socket\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.830208 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829412 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-run-ovn-kubernetes\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.830208 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829432 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/31e07d9c-93bc-4e8d-909b-e65406ff62df-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p9jxj\" (UID: \"31e07d9c-93bc-4e8d-909b-e65406ff62df\") " pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:17.830208 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829452 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-sys\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.830208 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829474 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-lib-modules\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.830208 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829498 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.830920 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829512 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-sys\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.830920 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829520 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cab63986-8b41-42c4-b7b0-4438e54fa424-host-slash\") pod \"iptables-alerter-q6grq\" (UID: \"cab63986-8b41-42c4-b7b0-4438e54fa424\") " pod="openshift-network-operator/iptables-alerter-q6grq" Nov 04 04:45:17.830920 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829543 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-sysconfig\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.830920 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829567 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-system-cni-dir\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.830920 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829584 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-lib-modules\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.830920 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829583 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-sysconfig\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.830920 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829592 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-cnibin\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.830920 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829613 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-system-cni-dir\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.830920 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829625 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-cnibin\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.830920 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829630 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-multus-socket-dir-parent\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.830920 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829655 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-host-var-lib-cni-multus\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.830920 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829673 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-multus-socket-dir-parent\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.830920 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829679 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/52f02c25-48ed-46dc-82bc-23016ffa595d-hosts-file\") pod \"node-resolver-7f22t\" (UID: \"52f02c25-48ed-46dc-82bc-23016ffa595d\") " pod="openshift-dns/node-resolver-7f22t" Nov 04 04:45:17.830920 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829701 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-host-var-lib-cni-multus\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.830920 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829706 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d9jw\" (UniqueName: \"kubernetes.io/projected/5c1153d6-83db-4eb7-95e0-78d09bd61d6d-kube-api-access-5d9jw\") pod \"node-ca-7tlqm\" (UID: \"5c1153d6-83db-4eb7-95e0-78d09bd61d6d\") " pod="openshift-image-registry/node-ca-7tlqm" Nov 04 04:45:17.830920 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829748 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-cni-binary-copy\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.830920 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829827 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-host-run-k8s-cni-cncf-io\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.830920 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829853 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-host-var-lib-kubelet\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.831522 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829876 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-etc-openvswitch\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.831522 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829881 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-host-run-k8s-cni-cncf-io\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.831522 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829895 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/31e07d9c-93bc-4e8d-909b-e65406ff62df-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p9jxj\" (UID: \"31e07d9c-93bc-4e8d-909b-e65406ff62df\") " pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:17.831522 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829904 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-modprobe-d\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.831522 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.829904 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-host-var-lib-kubelet\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.831522 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.830180 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-cni-binary-copy\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.831522 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.830496 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-modprobe-d\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.831522 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.830503 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-sysctl-d\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.831522 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.830532 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-run\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.831522 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.830554 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-multus-cni-dir\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.831522 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.830582 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/31e07d9c-93bc-4e8d-909b-e65406ff62df-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p9jxj\" (UID: \"31e07d9c-93bc-4e8d-909b-e65406ff62df\") " pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:17.831522 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.830604 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-sysctl-d\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.831522 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.830600 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-run\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.831522 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.830608 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-run-openvswitch\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.831522 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.830657 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-cni-bin\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.831522 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.830663 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-multus-cni-dir\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.831522 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.830676 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp26f\" (UniqueName: \"kubernetes.io/projected/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-kube-api-access-kp26f\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.831966 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.830728 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-registration-dir\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:17.831966 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.830754 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-cni-netd\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.831966 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.830787 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7380461b-403d-4470-8c1c-6433616b1dd2-env-overrides\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.831966 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.830816 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f02c25-48ed-46dc-82bc-23016ffa595d-tmp-dir\") pod \"node-resolver-7f22t\" (UID: \"52f02c25-48ed-46dc-82bc-23016ffa595d\") " pod="openshift-dns/node-resolver-7f22t" Nov 04 04:45:17.831966 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.830840 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7vxq\" (UniqueName: \"kubernetes.io/projected/52f02c25-48ed-46dc-82bc-23016ffa595d-kube-api-access-b7vxq\") pod \"node-resolver-7f22t\" (UID: \"52f02c25-48ed-46dc-82bc-23016ffa595d\") " pod="openshift-dns/node-resolver-7f22t" Nov 04 04:45:17.831966 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.831076 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/31e07d9c-93bc-4e8d-909b-e65406ff62df-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p9jxj\" (UID: \"31e07d9c-93bc-4e8d-909b-e65406ff62df\") " pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:17.831966 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.831797 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-tmp\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.831966 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.831851 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-tuned\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.854154 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.854137 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 04 04:45:17.854222 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.854181 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mkhmj" Nov 04 04:45:17.859285 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.859268 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 04 04:45:17.859381 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.859317 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 04 04:45:17.859831 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.859813 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 04 04:45:17.860116 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.860095 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 04 04:45:17.860116 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.860110 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 04 04:45:17.860381 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.860281 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-77ccl\"" Nov 04 04:45:17.860381 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.860373 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 04 04:45:17.860537 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.860524 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-95fm7\"" Nov 04 04:45:17.860715 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.860633 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 04 04:45:17.861005 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.860691 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 04 04:45:17.861086 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.860830 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kjpq6\"" Nov 04 04:45:17.861086 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.860951 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 04 04:45:17.861178 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.861125 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 04 04:45:17.861178 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.860956 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 04 04:45:17.896163 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:17.896118 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 04:45:17.896163 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:17.896137 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 04:45:17.896163 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:17.896147 2578 projected.go:196] Error preparing data for projected volume kube-api-access-b7jdp for pod openshift-network-diagnostics/network-check-target-mfn6w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 04:45:17.896330 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:17.896201 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59d18119-72a5-410f-be97-90613af651bd-kube-api-access-b7jdp podName:59d18119-72a5-410f-be97-90613af651bd nodeName:}" failed. No retries permitted until 2026-11-04 04:45:18.396186085 +0000 UTC m=+2.394754956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-b7jdp" (UniqueName: "kubernetes.io/projected/59d18119-72a5-410f-be97-90613af651bd-kube-api-access-b7jdp") pod "network-check-target-mfn6w" (UID: "59d18119-72a5-410f-be97-90613af651bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 04:45:17.898476 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.898456 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd5qw\" (UniqueName: \"kubernetes.io/projected/f0c260a1-2a73-4b2b-b6a7-2f9717941a1b-kube-api-access-pd5qw\") pod \"network-metrics-daemon-tkgd9\" (UID: \"f0c260a1-2a73-4b2b-b6a7-2f9717941a1b\") " pod="openshift-multus/network-metrics-daemon-tkgd9" Nov 04 04:45:17.898681 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.898666 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lppk7\" (UniqueName: \"kubernetes.io/projected/31e07d9c-93bc-4e8d-909b-e65406ff62df-kube-api-access-lppk7\") pod \"multus-additional-cni-plugins-p9jxj\" (UID: \"31e07d9c-93bc-4e8d-909b-e65406ff62df\") " pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:17.898877 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.898857 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6kb6\" (UniqueName: \"kubernetes.io/projected/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-kube-api-access-w6kb6\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:17.916370 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.916350 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d9jw\" (UniqueName: \"kubernetes.io/projected/5c1153d6-83db-4eb7-95e0-78d09bd61d6d-kube-api-access-5d9jw\") pod \"node-ca-7tlqm\" (UID: \"5c1153d6-83db-4eb7-95e0-78d09bd61d6d\") " pod="openshift-image-registry/node-ca-7tlqm" Nov 04 04:45:17.923912 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.923892 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp26f\" (UniqueName: \"kubernetes.io/projected/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-kube-api-access-kp26f\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:17.931336 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.931319 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-etc-openvswitch\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.931397 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.931346 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-run-openvswitch\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.931397 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.931360 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-cni-bin\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.931397 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.931375 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-registration-dir\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:17.931397 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.931392 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-cni-netd\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.931573 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.931413 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7380461b-403d-4470-8c1c-6433616b1dd2-env-overrides\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.931573 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.931436 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f02c25-48ed-46dc-82bc-23016ffa595d-tmp-dir\") pod \"node-resolver-7f22t\" (UID: \"52f02c25-48ed-46dc-82bc-23016ffa595d\") " pod="openshift-dns/node-resolver-7f22t" Nov 04 04:45:17.931573 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.931418 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-etc-openvswitch\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.931573 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.931462 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-cni-bin\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.931573 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.931464 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-registration-dir\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:17.931573 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.931466 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-cni-netd\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.931573 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.931484 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7vxq\" (UniqueName: \"kubernetes.io/projected/52f02c25-48ed-46dc-82bc-23016ffa595d-kube-api-access-b7vxq\") pod \"node-resolver-7f22t\" (UID: \"52f02c25-48ed-46dc-82bc-23016ffa595d\") " pod="openshift-dns/node-resolver-7f22t" Nov 04 04:45:17.931573 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.931512 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-run-openvswitch\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.931573 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.931524 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-node-log\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.931573 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.931553 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7380461b-403d-4470-8c1c-6433616b1dd2-ovnkube-config\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.932078 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.931594 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a7936867-0a26-4c2e-b94d-a1756cd14873-agent-certs\") pod \"konnectivity-agent-ct2mt\" (UID: \"a7936867-0a26-4c2e-b94d-a1756cd14873\") " pod="kube-system/konnectivity-agent-ct2mt" Nov 04 04:45:17.932078 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.931594 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-node-log\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.932078 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.931649 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gchdh\" (UniqueName: \"kubernetes.io/projected/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-kube-api-access-gchdh\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:17.932078 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.931694 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-device-dir\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:17.932078 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.931709 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-var-lib-openvswitch\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.932078 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.931724 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-run-ovn\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.932078 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.931743 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwmfd\" (UniqueName: \"kubernetes.io/projected/7380461b-403d-4470-8c1c-6433616b1dd2-kube-api-access-pwmfd\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.932078 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.931756 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cab63986-8b41-42c4-b7b0-4438e54fa424-iptables-alerter-script\") pod \"iptables-alerter-q6grq\" (UID: \"cab63986-8b41-42c4-b7b0-4438e54fa424\") " pod="openshift-network-operator/iptables-alerter-q6grq" Nov 04 04:45:17.932078 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.931769 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-etc-selinux\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:17.932078 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.931783 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-run-netns\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.932078 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.931799 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a7936867-0a26-4c2e-b94d-a1756cd14873-konnectivity-ca\") pod \"konnectivity-agent-ct2mt\" (UID: \"a7936867-0a26-4c2e-b94d-a1756cd14873\") " pod="kube-system/konnectivity-agent-ct2mt" Nov 04 04:45:17.932078 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.931814 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-run-ovn\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.932078 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.931823 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-systemd-units\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.932078 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.931876 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-run-netns\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.932078 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.931881 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-device-dir\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:17.932078 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.931929 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-var-lib-openvswitch\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.932078 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.931956 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zszmj\" (UniqueName: \"kubernetes.io/projected/cab63986-8b41-42c4-b7b0-4438e54fa424-kube-api-access-zszmj\") pod \"iptables-alerter-q6grq\" (UID: \"cab63986-8b41-42c4-b7b0-4438e54fa424\") " pod="openshift-network-operator/iptables-alerter-q6grq" Nov 04 04:45:17.932825 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.931974 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-etc-selinux\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:17.932825 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.931982 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7380461b-403d-4470-8c1c-6433616b1dd2-ovn-node-metrics-cert\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.932825 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.932002 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7380461b-403d-4470-8c1c-6433616b1dd2-ovnkube-script-lib\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.932825 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.932025 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:17.932825 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.932026 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-systemd-units\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.932825 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.932041 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-socket-dir\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:17.932825 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.932078 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-sys-fs\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:17.932825 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.932107 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-kubelet\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.932825 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.932127 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-slash\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.932825 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.932155 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:17.932825 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.932168 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-run-systemd\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.932825 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.932217 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-log-socket\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.932825 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.932229 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-slash\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.932825 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.932246 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-run-ovn-kubernetes\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.932825 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.932269 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-log-socket\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.932825 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.932277 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.932825 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.932323 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cab63986-8b41-42c4-b7b0-4438e54fa424-host-slash\") pod \"iptables-alerter-q6grq\" (UID: \"cab63986-8b41-42c4-b7b0-4438e54fa424\") " pod="openshift-network-operator/iptables-alerter-q6grq" Nov 04 04:45:17.933290 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.932337 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-run-systemd\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.933290 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.932353 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/52f02c25-48ed-46dc-82bc-23016ffa595d-hosts-file\") pod \"node-resolver-7f22t\" (UID: \"52f02c25-48ed-46dc-82bc-23016ffa595d\") " pod="openshift-dns/node-resolver-7f22t" Nov 04 04:45:17.933290 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.932373 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f02c25-48ed-46dc-82bc-23016ffa595d-tmp-dir\") pod \"node-resolver-7f22t\" (UID: \"52f02c25-48ed-46dc-82bc-23016ffa595d\") " pod="openshift-dns/node-resolver-7f22t" Nov 04 04:45:17.933290 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.932402 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-kubelet\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.933290 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.932409 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-socket-dir\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:17.933290 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.932435 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/52f02c25-48ed-46dc-82bc-23016ffa595d-hosts-file\") pod \"node-resolver-7f22t\" (UID: \"52f02c25-48ed-46dc-82bc-23016ffa595d\") " pod="openshift-dns/node-resolver-7f22t" Nov 04 04:45:17.933290 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.932235 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-sys-fs\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:17.933290 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.932459 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.933290 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.932454 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-run-ovn-kubernetes\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.933290 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.932519 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cab63986-8b41-42c4-b7b0-4438e54fa424-host-slash\") pod \"iptables-alerter-q6grq\" (UID: \"cab63986-8b41-42c4-b7b0-4438e54fa424\") " pod="openshift-network-operator/iptables-alerter-q6grq" Nov 04 04:45:17.933290 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.932672 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7380461b-403d-4470-8c1c-6433616b1dd2-env-overrides\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.933290 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.933272 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7380461b-403d-4470-8c1c-6433616b1dd2-ovnkube-config\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.933697 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.933356 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a7936867-0a26-4c2e-b94d-a1756cd14873-konnectivity-ca\") pod \"konnectivity-agent-ct2mt\" (UID: \"a7936867-0a26-4c2e-b94d-a1756cd14873\") " pod="kube-system/konnectivity-agent-ct2mt" Nov 04 04:45:17.933697 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.933434 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7380461b-403d-4470-8c1c-6433616b1dd2-ovnkube-script-lib\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:17.933697 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.933563 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cab63986-8b41-42c4-b7b0-4438e54fa424-iptables-alerter-script\") pod \"iptables-alerter-q6grq\" (UID: \"cab63986-8b41-42c4-b7b0-4438e54fa424\") " pod="openshift-network-operator/iptables-alerter-q6grq" Nov 04 04:45:17.934346 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.934327 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a7936867-0a26-4c2e-b94d-a1756cd14873-agent-certs\") pod \"konnectivity-agent-ct2mt\" (UID: \"a7936867-0a26-4c2e-b94d-a1756cd14873\") " pod="kube-system/konnectivity-agent-ct2mt" Nov 04 04:45:17.934443 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:17.934430 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7380461b-403d-4470-8c1c-6433616b1dd2-ovn-node-metrics-cert\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:18.000098 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:18.000075 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zszmj\" (UniqueName: \"kubernetes.io/projected/cab63986-8b41-42c4-b7b0-4438e54fa424-kube-api-access-zszmj\") pod \"iptables-alerter-q6grq\" (UID: \"cab63986-8b41-42c4-b7b0-4438e54fa424\") " pod="openshift-network-operator/iptables-alerter-q6grq" Nov 04 04:45:18.000278 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:18.000263 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwmfd\" (UniqueName: \"kubernetes.io/projected/7380461b-403d-4470-8c1c-6433616b1dd2-kube-api-access-pwmfd\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:18.000337 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:18.000319 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7vxq\" (UniqueName: \"kubernetes.io/projected/52f02c25-48ed-46dc-82bc-23016ffa595d-kube-api-access-b7vxq\") pod \"node-resolver-7f22t\" (UID: \"52f02c25-48ed-46dc-82bc-23016ffa595d\") " pod="openshift-dns/node-resolver-7f22t" Nov 04 04:45:18.000416 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:18.000401 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gchdh\" (UniqueName: \"kubernetes.io/projected/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-kube-api-access-gchdh\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:18.046063 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:18.046043 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:18.060940 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:18.060914 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7tlqm" Nov 04 04:45:18.080556 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:18.080539 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:18.086323 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:18.086293 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-m692r" Nov 04 04:45:18.096916 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:18.096897 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7f22t" Nov 04 04:45:18.125994 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:18.125974 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:18.144369 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:18.144348 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-q6grq" Nov 04 04:45:18.166777 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:18.166758 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ct2mt" Nov 04 04:45:18.170429 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:18.170412 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:18.234235 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:18.234213 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 04:45:18.243968 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:18.243937 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95a50811676d3438ddaa64148db61400.slice/crio-59837c93d67015f723085d6ae47d5e6e7c8a810f6bd904ef7641e6d6eee870d2 WatchSource:0}: Error finding container 59837c93d67015f723085d6ae47d5e6e7c8a810f6bd904ef7641e6d6eee870d2: Status 404 returned error can't find the container with id 59837c93d67015f723085d6ae47d5e6e7c8a810f6bd904ef7641e6d6eee870d2 Nov 04 04:45:18.244351 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:18.244332 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d814719e03f487d48b25be7946048f.slice/crio-e9f1723cdddd9a1bac5d41f731e8a1520baad29a0f2ccd976e47f907bf29486b WatchSource:0}: Error finding container e9f1723cdddd9a1bac5d41f731e8a1520baad29a0f2ccd976e47f907bf29486b: Status 404 returned error can't find the container with id e9f1723cdddd9a1bac5d41f731e8a1520baad29a0f2ccd976e47f907bf29486b Nov 04 04:45:18.249883 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:18.249867 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 04:45:18.334854 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:18.334832 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0c260a1-2a73-4b2b-b6a7-2f9717941a1b-metrics-certs\") pod \"network-metrics-daemon-tkgd9\" (UID: \"f0c260a1-2a73-4b2b-b6a7-2f9717941a1b\") " pod="openshift-multus/network-metrics-daemon-tkgd9" Nov 04 04:45:18.334934 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:18.334871 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3e3bf36f-7156-4151-9d6d-0937169dbcca-original-pull-secret\") pod \"global-pull-secret-syncer-mwgm5\" (UID: \"3e3bf36f-7156-4151-9d6d-0937169dbcca\") " pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:18.334983 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:18.334972 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 04:45:18.335017 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:18.334982 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 04:45:18.335017 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:18.335014 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e3bf36f-7156-4151-9d6d-0937169dbcca-original-pull-secret podName:3e3bf36f-7156-4151-9d6d-0937169dbcca nodeName:}" failed. No retries permitted until 2026-11-04 04:45:19.335001858 +0000 UTC m=+3.333570723 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3e3bf36f-7156-4151-9d6d-0937169dbcca-original-pull-secret") pod "global-pull-secret-syncer-mwgm5" (UID: "3e3bf36f-7156-4151-9d6d-0937169dbcca") : object "kube-system"/"original-pull-secret" not registered Nov 04 04:45:18.335087 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:18.335028 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0c260a1-2a73-4b2b-b6a7-2f9717941a1b-metrics-certs podName:f0c260a1-2a73-4b2b-b6a7-2f9717941a1b nodeName:}" failed. No retries permitted until 2026-11-04 04:45:19.33502122 +0000 UTC m=+3.333590072 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f0c260a1-2a73-4b2b-b6a7-2f9717941a1b-metrics-certs") pod "network-metrics-daemon-tkgd9" (UID: "f0c260a1-2a73-4b2b-b6a7-2f9717941a1b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 04:45:18.381826 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:18.381804 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 04:45:18.435377 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:18.435335 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7jdp\" (UniqueName: \"kubernetes.io/projected/59d18119-72a5-410f-be97-90613af651bd-kube-api-access-b7jdp\") pod \"network-check-target-mfn6w\" (UID: \"59d18119-72a5-410f-be97-90613af651bd\") " pod="openshift-network-diagnostics/network-check-target-mfn6w" Nov 04 04:45:18.435447 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:18.435414 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 04:45:18.435447 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:18.435429 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 04:45:18.435447 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:18.435440 2578 projected.go:196] Error preparing data for projected volume kube-api-access-b7jdp for pod openshift-network-diagnostics/network-check-target-mfn6w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 04:45:18.435559 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:18.435477 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59d18119-72a5-410f-be97-90613af651bd-kube-api-access-b7jdp podName:59d18119-72a5-410f-be97-90613af651bd nodeName:}" failed. No retries permitted until 2026-11-04 04:45:19.435465181 +0000 UTC m=+3.434034038 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-b7jdp" (UniqueName: "kubernetes.io/projected/59d18119-72a5-410f-be97-90613af651bd-kube-api-access-b7jdp") pod "network-check-target-mfn6w" (UID: "59d18119-72a5-410f-be97-90613af651bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 04:45:18.549217 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:18.549182 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-224.us-west-2.compute.internal" event={"ID":"88d814719e03f487d48b25be7946048f","Type":"ContainerStarted","Data":"e9f1723cdddd9a1bac5d41f731e8a1520baad29a0f2ccd976e47f907bf29486b"} Nov 04 04:45:18.550199 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:18.550178 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal" event={"ID":"95a50811676d3438ddaa64148db61400","Type":"ContainerStarted","Data":"59837c93d67015f723085d6ae47d5e6e7c8a810f6bd904ef7641e6d6eee870d2"} Nov 04 04:45:18.856678 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:18.856631 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 04:40:17 +0000 UTC" deadline="2028-04-29 09:12:11.267283228 +0000 UTC" Nov 04 04:45:18.856678 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:18.856651 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13012h26m52.410634321s" Nov 04 04:45:18.889769 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:18.889749 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae0b6cf_e85a_40a1_b61b_01bd5fde3643.slice/crio-9eeaf0c0b69ebb72502cadc763071934a68237a6e0eb623b96152a2f5fd0f6e4 WatchSource:0}: Error finding container 9eeaf0c0b69ebb72502cadc763071934a68237a6e0eb623b96152a2f5fd0f6e4: Status 404 returned error can't find the container with id 9eeaf0c0b69ebb72502cadc763071934a68237a6e0eb623b96152a2f5fd0f6e4 Nov 04 04:45:18.931677 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:18.931659 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 04:45:18.957364 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:18.957342 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c1153d6_83db_4eb7_95e0_78d09bd61d6d.slice/crio-24eecffe0d5a3df21a305e829689cc49da7665f450c7ac5084833a0a390efd7b WatchSource:0}: Error finding container 24eecffe0d5a3df21a305e829689cc49da7665f450c7ac5084833a0a390efd7b: Status 404 returned error can't find the container with id 24eecffe0d5a3df21a305e829689cc49da7665f450c7ac5084833a0a390efd7b Nov 04 04:45:18.963649 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:18.963630 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52f02c25_48ed_46dc_82bc_23016ffa595d.slice/crio-da6f323aee2ca34cdfc7d0e4609ca9d1830c997a3d52cdb2b1cacbdb73886064 WatchSource:0}: Error finding container da6f323aee2ca34cdfc7d0e4609ca9d1830c997a3d52cdb2b1cacbdb73886064: Status 404 returned error can't find the container with id da6f323aee2ca34cdfc7d0e4609ca9d1830c997a3d52cdb2b1cacbdb73886064 Nov 04 04:45:18.992128 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:18.992109 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc352ca6_4a2d_4f7f_a434_7c45dd704e4b.slice/crio-5ca6e5771579d93b64701a87510bbf7d2d64d3e6cf58bb36c4633844df0a5081 WatchSource:0}: Error finding container 5ca6e5771579d93b64701a87510bbf7d2d64d3e6cf58bb36c4633844df0a5081: Status 404 returned error can't find the container with id 5ca6e5771579d93b64701a87510bbf7d2d64d3e6cf58bb36c4633844df0a5081 Nov 04 04:45:19.064939 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:19.064921 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31e07d9c_93bc_4e8d_909b_e65406ff62df.slice/crio-23b43177dc4829aaf4c28707186c6ac0c41d50126821789ce3dd9cae71a77721 WatchSource:0}: Error finding container 23b43177dc4829aaf4c28707186c6ac0c41d50126821789ce3dd9cae71a77721: Status 404 returned error can't find the container with id 23b43177dc4829aaf4c28707186c6ac0c41d50126821789ce3dd9cae71a77721 Nov 04 04:45:19.070245 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:19.070223 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcab63986_8b41_42c4_b7b0_4438e54fa424.slice/crio-9c5ddfa0f2d283e32032e690290a63a1f13dafec60cc2bdac506c6980248ff45 WatchSource:0}: Error finding container 9c5ddfa0f2d283e32032e690290a63a1f13dafec60cc2bdac506c6980248ff45: Status 404 returned error can't find the container with id 9c5ddfa0f2d283e32032e690290a63a1f13dafec60cc2bdac506c6980248ff45 Nov 04 04:45:19.094920 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:19.094898 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7380461b_403d_4470_8c1c_6433616b1dd2.slice/crio-c0105ca9ce2f010d642378a9904480ecf73c2953090017a5d23f612c5aea388f WatchSource:0}: Error finding container c0105ca9ce2f010d642378a9904480ecf73c2953090017a5d23f612c5aea388f: Status 404 returned error can't find the container with id c0105ca9ce2f010d642378a9904480ecf73c2953090017a5d23f612c5aea388f Nov 04 04:45:19.120049 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:19.120030 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7936867_0a26_4c2e_b94d_a1756cd14873.slice/crio-dc7d6e11710a5c4826009a9c64aaa572ca406dedd7ab40a50497e27f7bfa3004 WatchSource:0}: Error finding container dc7d6e11710a5c4826009a9c64aaa572ca406dedd7ab40a50497e27f7bfa3004: Status 404 returned error can't find the container with id dc7d6e11710a5c4826009a9c64aaa572ca406dedd7ab40a50497e27f7bfa3004 Nov 04 04:45:19.341432 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:19.341411 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3e3bf36f-7156-4151-9d6d-0937169dbcca-original-pull-secret\") pod \"global-pull-secret-syncer-mwgm5\" (UID: \"3e3bf36f-7156-4151-9d6d-0937169dbcca\") " pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:19.341540 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:19.341467 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0c260a1-2a73-4b2b-b6a7-2f9717941a1b-metrics-certs\") pod \"network-metrics-daemon-tkgd9\" (UID: \"f0c260a1-2a73-4b2b-b6a7-2f9717941a1b\") " pod="openshift-multus/network-metrics-daemon-tkgd9" Nov 04 04:45:19.341604 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:19.341548 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 04:45:19.341604 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:19.341581 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 04:45:19.341692 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:19.341605 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e3bf36f-7156-4151-9d6d-0937169dbcca-original-pull-secret podName:3e3bf36f-7156-4151-9d6d-0937169dbcca nodeName:}" failed. No retries permitted until 2026-11-04 04:45:21.341587957 +0000 UTC m=+5.340156818 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3e3bf36f-7156-4151-9d6d-0937169dbcca-original-pull-secret") pod "global-pull-secret-syncer-mwgm5" (UID: "3e3bf36f-7156-4151-9d6d-0937169dbcca") : object "kube-system"/"original-pull-secret" not registered Nov 04 04:45:19.341692 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:19.341627 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0c260a1-2a73-4b2b-b6a7-2f9717941a1b-metrics-certs podName:f0c260a1-2a73-4b2b-b6a7-2f9717941a1b nodeName:}" failed. No retries permitted until 2026-11-04 04:45:21.341612606 +0000 UTC m=+5.340181460 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f0c260a1-2a73-4b2b-b6a7-2f9717941a1b-metrics-certs") pod "network-metrics-daemon-tkgd9" (UID: "f0c260a1-2a73-4b2b-b6a7-2f9717941a1b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 04:45:19.386530 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:19.386512 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7422e4c1_59b2_4bdc_8fb1_7542ba692a6a.slice/crio-52b6671ae663b31c624bd621cb13d966ee18207be2a20611890ab2f9c1b70b08 WatchSource:0}: Error finding container 52b6671ae663b31c624bd621cb13d966ee18207be2a20611890ab2f9c1b70b08: Status 404 returned error can't find the container with id 52b6671ae663b31c624bd621cb13d966ee18207be2a20611890ab2f9c1b70b08 Nov 04 04:45:19.442001 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:19.441974 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7jdp\" (UniqueName: \"kubernetes.io/projected/59d18119-72a5-410f-be97-90613af651bd-kube-api-access-b7jdp\") pod \"network-check-target-mfn6w\" (UID: \"59d18119-72a5-410f-be97-90613af651bd\") " pod="openshift-network-diagnostics/network-check-target-mfn6w" Nov 04 04:45:19.442106 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:19.442097 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 04:45:19.442155 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:19.442113 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 04:45:19.442155 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:19.442122 2578 projected.go:196] Error preparing data for projected volume kube-api-access-b7jdp for pod openshift-network-diagnostics/network-check-target-mfn6w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 04:45:19.442238 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:19.442157 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59d18119-72a5-410f-be97-90613af651bd-kube-api-access-b7jdp podName:59d18119-72a5-410f-be97-90613af651bd nodeName:}" failed. No retries permitted until 2026-11-04 04:45:21.442145696 +0000 UTC m=+5.440714562 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-b7jdp" (UniqueName: "kubernetes.io/projected/59d18119-72a5-410f-be97-90613af651bd-kube-api-access-b7jdp") pod "network-check-target-mfn6w" (UID: "59d18119-72a5-410f-be97-90613af651bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 04:45:19.546585 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:19.546562 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfn6w" Nov 04 04:45:19.546714 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:19.546602 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:19.546714 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:19.546664 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfn6w" podUID="59d18119-72a5-410f-be97-90613af651bd" Nov 04 04:45:19.546714 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:19.546687 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkgd9" Nov 04 04:45:19.546865 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:19.546751 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwgm5" podUID="3e3bf36f-7156-4151-9d6d-0937169dbcca" Nov 04 04:45:19.546865 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:19.546811 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkgd9" podUID="f0c260a1-2a73-4b2b-b6a7-2f9717941a1b" Nov 04 04:45:19.552754 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:19.552731 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" event={"ID":"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a","Type":"ContainerStarted","Data":"52b6671ae663b31c624bd621cb13d966ee18207be2a20611890ab2f9c1b70b08"} Nov 04 04:45:19.554073 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:19.554050 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" event={"ID":"7380461b-403d-4470-8c1c-6433616b1dd2","Type":"ContainerStarted","Data":"c0105ca9ce2f010d642378a9904480ecf73c2953090017a5d23f612c5aea388f"} Nov 04 04:45:19.556112 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:19.556092 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-q6grq" event={"ID":"cab63986-8b41-42c4-b7b0-4438e54fa424","Type":"ContainerStarted","Data":"9c5ddfa0f2d283e32032e690290a63a1f13dafec60cc2bdac506c6980248ff45"} Nov 04 04:45:19.557350 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:19.557328 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p9jxj" event={"ID":"31e07d9c-93bc-4e8d-909b-e65406ff62df","Type":"ContainerStarted","Data":"23b43177dc4829aaf4c28707186c6ac0c41d50126821789ce3dd9cae71a77721"} Nov 04 04:45:19.558644 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:19.558611 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7tlqm" event={"ID":"5c1153d6-83db-4eb7-95e0-78d09bd61d6d","Type":"ContainerStarted","Data":"24eecffe0d5a3df21a305e829689cc49da7665f450c7ac5084833a0a390efd7b"} Nov 04 04:45:19.560061 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:19.560040 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ct2mt" event={"ID":"a7936867-0a26-4c2e-b94d-a1756cd14873","Type":"ContainerStarted","Data":"dc7d6e11710a5c4826009a9c64aaa572ca406dedd7ab40a50497e27f7bfa3004"} Nov 04 04:45:19.561700 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:19.561670 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-m692r" event={"ID":"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b","Type":"ContainerStarted","Data":"5ca6e5771579d93b64701a87510bbf7d2d64d3e6cf58bb36c4633844df0a5081"} Nov 04 04:45:19.563157 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:19.563134 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7f22t" event={"ID":"52f02c25-48ed-46dc-82bc-23016ffa595d","Type":"ContainerStarted","Data":"da6f323aee2ca34cdfc7d0e4609ca9d1830c997a3d52cdb2b1cacbdb73886064"} Nov 04 04:45:19.565743 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:19.565720 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" event={"ID":"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643","Type":"ContainerStarted","Data":"9eeaf0c0b69ebb72502cadc763071934a68237a6e0eb623b96152a2f5fd0f6e4"} Nov 04 04:45:19.857660 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:19.857559 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 04:40:17 +0000 UTC" deadline="2028-07-02 03:16:25.973675683 +0000 UTC" Nov 04 04:45:19.857660 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:19.857614 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14542h31m6.11606641s" Nov 04 04:45:21.357541 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:21.357133 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0c260a1-2a73-4b2b-b6a7-2f9717941a1b-metrics-certs\") pod \"network-metrics-daemon-tkgd9\" (UID: \"f0c260a1-2a73-4b2b-b6a7-2f9717941a1b\") " pod="openshift-multus/network-metrics-daemon-tkgd9" Nov 04 04:45:21.357541 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:21.357203 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3e3bf36f-7156-4151-9d6d-0937169dbcca-original-pull-secret\") pod \"global-pull-secret-syncer-mwgm5\" (UID: \"3e3bf36f-7156-4151-9d6d-0937169dbcca\") " pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:21.357541 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:21.357352 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 04:45:21.357541 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:21.357407 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e3bf36f-7156-4151-9d6d-0937169dbcca-original-pull-secret podName:3e3bf36f-7156-4151-9d6d-0937169dbcca nodeName:}" failed. No retries permitted until 2026-11-04 04:45:25.357390364 +0000 UTC m=+9.355959221 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3e3bf36f-7156-4151-9d6d-0937169dbcca-original-pull-secret") pod "global-pull-secret-syncer-mwgm5" (UID: "3e3bf36f-7156-4151-9d6d-0937169dbcca") : object "kube-system"/"original-pull-secret" not registered Nov 04 04:45:21.357541 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:21.357472 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 04:45:21.357541 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:21.357506 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0c260a1-2a73-4b2b-b6a7-2f9717941a1b-metrics-certs podName:f0c260a1-2a73-4b2b-b6a7-2f9717941a1b nodeName:}" failed. No retries permitted until 2026-11-04 04:45:25.357494117 +0000 UTC m=+9.356062984 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f0c260a1-2a73-4b2b-b6a7-2f9717941a1b-metrics-certs") pod "network-metrics-daemon-tkgd9" (UID: "f0c260a1-2a73-4b2b-b6a7-2f9717941a1b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 04:45:21.460845 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:21.460552 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7jdp\" (UniqueName: \"kubernetes.io/projected/59d18119-72a5-410f-be97-90613af651bd-kube-api-access-b7jdp\") pod \"network-check-target-mfn6w\" (UID: \"59d18119-72a5-410f-be97-90613af651bd\") " pod="openshift-network-diagnostics/network-check-target-mfn6w" Nov 04 04:45:21.460845 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:21.460733 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 04:45:21.460845 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:21.460749 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 04:45:21.460845 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:21.460761 2578 projected.go:196] Error preparing data for projected volume kube-api-access-b7jdp for pod openshift-network-diagnostics/network-check-target-mfn6w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 04:45:21.460845 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:21.460809 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59d18119-72a5-410f-be97-90613af651bd-kube-api-access-b7jdp podName:59d18119-72a5-410f-be97-90613af651bd nodeName:}" failed. No retries permitted until 2026-11-04 04:45:25.460793386 +0000 UTC m=+9.459362254 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-b7jdp" (UniqueName: "kubernetes.io/projected/59d18119-72a5-410f-be97-90613af651bd-kube-api-access-b7jdp") pod "network-check-target-mfn6w" (UID: "59d18119-72a5-410f-be97-90613af651bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 04:45:21.546727 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:21.546667 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkgd9" Nov 04 04:45:21.546854 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:21.546797 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkgd9" podUID="f0c260a1-2a73-4b2b-b6a7-2f9717941a1b" Nov 04 04:45:21.546910 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:21.546869 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfn6w" Nov 04 04:45:21.546968 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:21.546944 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfn6w" podUID="59d18119-72a5-410f-be97-90613af651bd" Nov 04 04:45:21.547049 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:21.547009 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:21.547100 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:21.547085 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwgm5" podUID="3e3bf36f-7156-4151-9d6d-0937169dbcca" Nov 04 04:45:22.594243 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:22.593995 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-224.us-west-2.compute.internal" event={"ID":"88d814719e03f487d48b25be7946048f","Type":"ContainerStarted","Data":"e198945739bceb0aa79e60e26855d5dbabb1932df5eaeddb37c371626682dd03"} Nov 04 04:45:22.599084 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:22.598550 2578 generic.go:356] "Generic (PLEG): container finished" podID="95a50811676d3438ddaa64148db61400" containerID="50f109a8342c11809dea05e856eb8ec071a03f0979c626a99e4dedb7fcd3e0da" exitCode=0 Nov 04 04:45:22.599084 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:22.598587 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal" event={"ID":"95a50811676d3438ddaa64148db61400","Type":"ContainerDied","Data":"50f109a8342c11809dea05e856eb8ec071a03f0979c626a99e4dedb7fcd3e0da"} Nov 04 04:45:22.854621 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:22.853765 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-224.us-west-2.compute.internal" podStartSLOduration=6.85374663 podStartE2EDuration="6.85374663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 04:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:45:22.723650436 +0000 UTC m=+6.722219310" watchObservedRunningTime="2026-11-04 04:45:22.85374663 +0000 UTC m=+6.852315507" Nov 04 04:45:23.546409 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:23.546378 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:23.546579 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:23.546503 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwgm5" podUID="3e3bf36f-7156-4151-9d6d-0937169dbcca" Nov 04 04:45:23.546646 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:23.546582 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkgd9" Nov 04 04:45:23.546699 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:23.546659 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkgd9" podUID="f0c260a1-2a73-4b2b-b6a7-2f9717941a1b" Nov 04 04:45:23.546758 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:23.546704 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfn6w" Nov 04 04:45:23.546803 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:23.546761 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfn6w" podUID="59d18119-72a5-410f-be97-90613af651bd" Nov 04 04:45:23.605659 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:23.605628 2578 generic.go:356] "Generic (PLEG): container finished" podID="31e07d9c-93bc-4e8d-909b-e65406ff62df" containerID="d72d5c0598318327eaad79bb5dcd047d76f14c5f9f8287eb87c43e68ad3ec5f1" exitCode=0 Nov 04 04:45:23.606060 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:23.605723 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p9jxj" event={"ID":"31e07d9c-93bc-4e8d-909b-e65406ff62df","Type":"ContainerDied","Data":"d72d5c0598318327eaad79bb5dcd047d76f14c5f9f8287eb87c43e68ad3ec5f1"} Nov 04 04:45:23.618183 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:23.618086 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal" event={"ID":"95a50811676d3438ddaa64148db61400","Type":"ContainerStarted","Data":"aeb3f05d45aebc2e4a633045094aca5234ea1ce5006567bae982477fd1f56dbe"} Nov 04 04:45:24.621878 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:24.621803 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7tlqm" event={"ID":"5c1153d6-83db-4eb7-95e0-78d09bd61d6d","Type":"ContainerStarted","Data":"e479ee445948ef9270b342c45e2ff03f28b6319fdb6a31bee61e2b9ba136ff09"} Nov 04 04:45:24.627115 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:24.627067 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ct2mt" event={"ID":"a7936867-0a26-4c2e-b94d-a1756cd14873","Type":"ContainerStarted","Data":"8e52ae7e9192f2b4a0caf15eb42e9a64d533143dc955f7161fc2a6d2e5776d86"} Nov 04 04:45:24.752438 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:24.752347 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7tlqm" podStartSLOduration=3.356551284 podStartE2EDuration="8.752328704s" totalImagesPullingTime="5.39577742s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 04:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:45:24.752029639 +0000 UTC m=+8.750598512" watchObservedRunningTime="2026-11-04 04:45:24.752328704 +0000 UTC m=+8.750897571" Nov 04 04:45:24.752618 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:24.752477 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal" podStartSLOduration=8.752471275 podStartE2EDuration="8.752471275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 04:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:45:23.876191479 +0000 UTC m=+7.874760355" watchObservedRunningTime="2026-11-04 04:45:24.752471275 +0000 UTC m=+8.751040149" Nov 04 04:45:24.882479 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:24.882386 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ct2mt" podStartSLOduration=4.440494502 podStartE2EDuration="8.882366325s" totalImagesPullingTime="4.441871823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:45:24.881843164 +0000 UTC m=+8.880412038" watchObservedRunningTime="2026-11-04 04:45:24.882366325 +0000 UTC m=+8.880935200" Nov 04 04:45:25.393845 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:25.393814 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0c260a1-2a73-4b2b-b6a7-2f9717941a1b-metrics-certs\") pod \"network-metrics-daemon-tkgd9\" (UID: \"f0c260a1-2a73-4b2b-b6a7-2f9717941a1b\") " pod="openshift-multus/network-metrics-daemon-tkgd9" Nov 04 04:45:25.394011 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:25.393879 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3e3bf36f-7156-4151-9d6d-0937169dbcca-original-pull-secret\") pod \"global-pull-secret-syncer-mwgm5\" (UID: \"3e3bf36f-7156-4151-9d6d-0937169dbcca\") " pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:25.394011 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:25.393975 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 04:45:25.394140 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:25.394018 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e3bf36f-7156-4151-9d6d-0937169dbcca-original-pull-secret podName:3e3bf36f-7156-4151-9d6d-0937169dbcca nodeName:}" failed. No retries permitted until 2026-11-04 04:45:33.394004886 +0000 UTC m=+17.392573751 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3e3bf36f-7156-4151-9d6d-0937169dbcca-original-pull-secret") pod "global-pull-secret-syncer-mwgm5" (UID: "3e3bf36f-7156-4151-9d6d-0937169dbcca") : object "kube-system"/"original-pull-secret" not registered Nov 04 04:45:25.394140 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:25.394060 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 04:45:25.394140 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:25.394079 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0c260a1-2a73-4b2b-b6a7-2f9717941a1b-metrics-certs podName:f0c260a1-2a73-4b2b-b6a7-2f9717941a1b nodeName:}" failed. No retries permitted until 2026-11-04 04:45:33.39407268 +0000 UTC m=+17.392641532 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f0c260a1-2a73-4b2b-b6a7-2f9717941a1b-metrics-certs") pod "network-metrics-daemon-tkgd9" (UID: "f0c260a1-2a73-4b2b-b6a7-2f9717941a1b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 04:45:25.494557 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:25.494512 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7jdp\" (UniqueName: \"kubernetes.io/projected/59d18119-72a5-410f-be97-90613af651bd-kube-api-access-b7jdp\") pod \"network-check-target-mfn6w\" (UID: \"59d18119-72a5-410f-be97-90613af651bd\") " pod="openshift-network-diagnostics/network-check-target-mfn6w" Nov 04 04:45:25.494690 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:25.494668 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 04:45:25.494690 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:25.494688 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 04:45:25.494798 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:25.494698 2578 projected.go:196] Error preparing data for projected volume kube-api-access-b7jdp for pod openshift-network-diagnostics/network-check-target-mfn6w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 04:45:25.494798 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:25.494734 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59d18119-72a5-410f-be97-90613af651bd-kube-api-access-b7jdp podName:59d18119-72a5-410f-be97-90613af651bd nodeName:}" failed. No retries permitted until 2026-11-04 04:45:33.49472281 +0000 UTC m=+17.493291686 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-b7jdp" (UniqueName: "kubernetes.io/projected/59d18119-72a5-410f-be97-90613af651bd-kube-api-access-b7jdp") pod "network-check-target-mfn6w" (UID: "59d18119-72a5-410f-be97-90613af651bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 04:45:25.546433 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:25.546407 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfn6w" Nov 04 04:45:25.546555 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:25.546529 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfn6w" podUID="59d18119-72a5-410f-be97-90613af651bd" Nov 04 04:45:25.546604 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:25.546579 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:25.546661 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:25.546643 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwgm5" podUID="3e3bf36f-7156-4151-9d6d-0937169dbcca" Nov 04 04:45:25.546712 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:25.546692 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkgd9" Nov 04 04:45:25.546782 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:25.546765 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkgd9" podUID="f0c260a1-2a73-4b2b-b6a7-2f9717941a1b" Nov 04 04:45:26.636895 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:26.636855 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" event={"ID":"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a","Type":"ContainerStarted","Data":"91df5c326a99b139e025ff2dd6d99b8daa0faeff4205c286fd4e5d86626dc83e"} Nov 04 04:45:27.546708 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:27.546231 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfn6w" Nov 04 04:45:27.546708 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:27.546367 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfn6w" podUID="59d18119-72a5-410f-be97-90613af651bd" Nov 04 04:45:27.546708 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:27.546441 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:27.546708 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:27.546511 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwgm5" podUID="3e3bf36f-7156-4151-9d6d-0937169dbcca" Nov 04 04:45:27.546708 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:27.546570 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkgd9" Nov 04 04:45:27.546708 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:27.546667 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkgd9" podUID="f0c260a1-2a73-4b2b-b6a7-2f9717941a1b" Nov 04 04:45:27.640829 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:27.640201 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-m692r" event={"ID":"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b","Type":"ContainerStarted","Data":"9a1dbb94208a115b43b7ee72778122bcdd03a495a3363eab97de64c22cbbac85"} Nov 04 04:45:27.860982 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:27.860912 2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 04 04:45:28.481727 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:28.481619 2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-04T04:45:27.860932766Z","UUID":"b7a75198-8039-40d0-bcff-b619b7922578","Handler":null,"Name":"","Endpoint":""} Nov 04 04:45:28.483256 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:28.483231 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 04 04:45:28.483256 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:28.483262 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 04 04:45:28.644499 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:28.644429 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7f22t" event={"ID":"52f02c25-48ed-46dc-82bc-23016ffa595d","Type":"ContainerStarted","Data":"06d415d402e00841205f22197e3ec072ac0d24efe7446b23c1b58dc0176c31ff"} Nov 04 04:45:28.646832 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:28.646802 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" event={"ID":"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a","Type":"ContainerStarted","Data":"d79cb596395360567e913807660a189b7eeba93b123482eacecaf95d9570b1e2"} Nov 04 04:45:28.649226 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:28.649196 2578 generic.go:356] "Generic (PLEG): container finished" podID="31e07d9c-93bc-4e8d-909b-e65406ff62df" containerID="5369c00880cb269e292e9a796b5eed7b98e49708007bb73a970daf58ffde2caf" exitCode=0 Nov 04 04:45:28.649341 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:28.649254 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p9jxj" event={"ID":"31e07d9c-93bc-4e8d-909b-e65406ff62df","Type":"ContainerDied","Data":"5369c00880cb269e292e9a796b5eed7b98e49708007bb73a970daf58ffde2caf"} Nov 04 04:45:28.774638 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:28.774580 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-m692r" podStartSLOduration=4.592084512 podStartE2EDuration="12.774563881s" totalImagesPullingTime="8.182479369s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 04:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:45:27.774971536 +0000 UTC m=+11.773540419" watchObservedRunningTime="2026-11-04 04:45:28.774563881 +0000 UTC m=+12.773132757" Nov 04 04:45:28.774947 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:28.774916 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-7f22t" podStartSLOduration=4.099920608 podStartE2EDuration="12.774903893s" totalImagesPullingTime="8.674983285s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 04:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:45:28.774165554 +0000 UTC m=+12.772734453" watchObservedRunningTime="2026-11-04 04:45:28.774903893 +0000 UTC m=+12.773472768" Nov 04 04:45:28.871701 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:28.871672 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ct2mt" Nov 04 04:45:28.872888 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:28.872868 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ct2mt" Nov 04 04:45:29.325846 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:29.325630 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ct2mt" Nov 04 04:45:29.326540 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:29.326517 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ct2mt" Nov 04 04:45:29.546994 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:29.546968 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfn6w" Nov 04 04:45:29.547115 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:29.547007 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:29.547115 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:29.547062 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkgd9" Nov 04 04:45:29.547115 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:29.547077 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfn6w" podUID="59d18119-72a5-410f-be97-90613af651bd" Nov 04 04:45:29.547254 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:29.547140 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwgm5" podUID="3e3bf36f-7156-4151-9d6d-0937169dbcca" Nov 04 04:45:29.547254 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:29.547205 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkgd9" podUID="f0c260a1-2a73-4b2b-b6a7-2f9717941a1b" Nov 04 04:45:29.654055 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:29.653993 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" event={"ID":"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a","Type":"ContainerStarted","Data":"97ca0012fb7980372b02d2ec8d2e95bf83eef28644013683040b37ec84e42c09"} Nov 04 04:45:29.655563 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:29.655540 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-q6grq" event={"ID":"cab63986-8b41-42c4-b7b0-4438e54fa424","Type":"ContainerStarted","Data":"44b4efcd27b5b028e5f1219ba83e545cc57cf152ce8c6e7f4169b737063f41e2"} Nov 04 04:45:29.657652 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:29.657629 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" event={"ID":"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643","Type":"ContainerStarted","Data":"f26e579fc0afd09d8c130a65a3ac131ec5558c5bb28cd03892a08b1c36f5b5bc"} Nov 04 04:45:29.787509 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:29.787468 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" podStartSLOduration=4.117165717 podStartE2EDuration="13.787452573s" totalImagesPullingTime="9.670286856s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 04:45:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:45:29.787412516 +0000 UTC m=+13.785981391" watchObservedRunningTime="2026-11-04 04:45:29.787452573 +0000 UTC m=+13.786021448" Nov 04 04:45:29.916567 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:29.916529 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-q6grq" podStartSLOduration=5.503934736 podStartE2EDuration="13.916516337s" totalImagesPullingTime="8.412581601s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 04:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:45:29.916512934 +0000 UTC m=+13.915081812" watchObservedRunningTime="2026-11-04 04:45:29.916516337 +0000 UTC m=+13.915085220" Nov 04 04:45:30.049828 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:30.049783 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" podStartSLOduration=3.680043548 podStartE2EDuration="14.049770257s" totalImagesPullingTime="10.369726709s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 04:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:45:30.049746936 +0000 UTC m=+14.048315811" watchObservedRunningTime="2026-11-04 04:45:30.049770257 +0000 UTC m=+14.048339121" Nov 04 04:45:30.662084 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:30.662053 2578 generic.go:356] "Generic (PLEG): container finished" podID="31e07d9c-93bc-4e8d-909b-e65406ff62df" containerID="f1ed44958c2fa0725dcb552eb33bc9cdf5083cf216adc03e079e32fdd8a1afcd" exitCode=0 Nov 04 04:45:30.662571 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:30.662155 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p9jxj" event={"ID":"31e07d9c-93bc-4e8d-909b-e65406ff62df","Type":"ContainerDied","Data":"f1ed44958c2fa0725dcb552eb33bc9cdf5083cf216adc03e079e32fdd8a1afcd"} Nov 04 04:45:31.546873 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:31.546841 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkgd9" Nov 04 04:45:31.547046 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:31.546882 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:31.547046 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:31.546889 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfn6w" Nov 04 04:45:31.547046 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:31.546970 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkgd9" podUID="f0c260a1-2a73-4b2b-b6a7-2f9717941a1b" Nov 04 04:45:31.547046 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:31.547020 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwgm5" podUID="3e3bf36f-7156-4151-9d6d-0937169dbcca" Nov 04 04:45:31.547188 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:31.547096 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfn6w" podUID="59d18119-72a5-410f-be97-90613af651bd" Nov 04 04:45:32.668954 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:32.668918 2578 generic.go:356] "Generic (PLEG): container finished" podID="31e07d9c-93bc-4e8d-909b-e65406ff62df" containerID="486a127530789d1f203e7868d4b26167e4e57bef4f14da888832d7cbe477efa3" exitCode=0 Nov 04 04:45:32.669762 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:32.668961 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p9jxj" event={"ID":"31e07d9c-93bc-4e8d-909b-e65406ff62df","Type":"ContainerDied","Data":"486a127530789d1f203e7868d4b26167e4e57bef4f14da888832d7cbe477efa3"} Nov 04 04:45:33.451383 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:33.451353 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3e3bf36f-7156-4151-9d6d-0937169dbcca-original-pull-secret\") pod \"global-pull-secret-syncer-mwgm5\" (UID: \"3e3bf36f-7156-4151-9d6d-0937169dbcca\") " pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:33.451559 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:33.451414 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0c260a1-2a73-4b2b-b6a7-2f9717941a1b-metrics-certs\") pod \"network-metrics-daemon-tkgd9\" (UID: \"f0c260a1-2a73-4b2b-b6a7-2f9717941a1b\") " pod="openshift-multus/network-metrics-daemon-tkgd9" Nov 04 04:45:33.451559 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:33.451493 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 04:45:33.451559 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:33.451518 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 04:45:33.451559 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:33.451559 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e3bf36f-7156-4151-9d6d-0937169dbcca-original-pull-secret podName:3e3bf36f-7156-4151-9d6d-0937169dbcca nodeName:}" failed. No retries permitted until 2026-11-04 04:45:49.451541356 +0000 UTC m=+33.450110209 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3e3bf36f-7156-4151-9d6d-0937169dbcca-original-pull-secret") pod "global-pull-secret-syncer-mwgm5" (UID: "3e3bf36f-7156-4151-9d6d-0937169dbcca") : object "kube-system"/"original-pull-secret" not registered Nov 04 04:45:33.451768 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:33.451576 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0c260a1-2a73-4b2b-b6a7-2f9717941a1b-metrics-certs podName:f0c260a1-2a73-4b2b-b6a7-2f9717941a1b nodeName:}" failed. No retries permitted until 2026-11-04 04:45:49.451569105 +0000 UTC m=+33.450137958 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f0c260a1-2a73-4b2b-b6a7-2f9717941a1b-metrics-certs") pod "network-metrics-daemon-tkgd9" (UID: "f0c260a1-2a73-4b2b-b6a7-2f9717941a1b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 04:45:33.546825 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:33.546780 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:33.546825 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:33.546801 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkgd9" Nov 04 04:45:33.547047 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:33.546910 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfn6w" Nov 04 04:45:33.547047 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:33.546911 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwgm5" podUID="3e3bf36f-7156-4151-9d6d-0937169dbcca" Nov 04 04:45:33.547047 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:33.546996 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkgd9" podUID="f0c260a1-2a73-4b2b-b6a7-2f9717941a1b" Nov 04 04:45:33.547197 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:33.547069 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfn6w" podUID="59d18119-72a5-410f-be97-90613af651bd" Nov 04 04:45:33.551910 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:33.551885 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7jdp\" (UniqueName: \"kubernetes.io/projected/59d18119-72a5-410f-be97-90613af651bd-kube-api-access-b7jdp\") pod \"network-check-target-mfn6w\" (UID: \"59d18119-72a5-410f-be97-90613af651bd\") " pod="openshift-network-diagnostics/network-check-target-mfn6w" Nov 04 04:45:33.552096 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:33.552012 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 04:45:33.552096 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:33.552030 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 04:45:33.552096 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:33.552040 2578 projected.go:196] Error preparing data for projected volume kube-api-access-b7jdp for pod openshift-network-diagnostics/network-check-target-mfn6w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 04:45:33.552096 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:33.552077 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59d18119-72a5-410f-be97-90613af651bd-kube-api-access-b7jdp podName:59d18119-72a5-410f-be97-90613af651bd nodeName:}" failed. No retries permitted until 2026-11-04 04:45:49.552066324 +0000 UTC m=+33.550635176 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-b7jdp" (UniqueName: "kubernetes.io/projected/59d18119-72a5-410f-be97-90613af651bd-kube-api-access-b7jdp") pod "network-check-target-mfn6w" (UID: "59d18119-72a5-410f-be97-90613af651bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 04:45:34.676933 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:34.676579 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" event={"ID":"7380461b-403d-4470-8c1c-6433616b1dd2","Type":"ContainerStarted","Data":"ac357af3e6c86470018be9df4195917cb95dc6ffd06185bd8a377eb08171e581"} Nov 04 04:45:34.677369 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:34.676963 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" event={"ID":"7380461b-403d-4470-8c1c-6433616b1dd2","Type":"ContainerStarted","Data":"8051cac011c970f816659b2b908fbe4d50e540f74395e916aad3e58bd0429136"} Nov 04 04:45:34.677369 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:34.677010 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" event={"ID":"7380461b-403d-4470-8c1c-6433616b1dd2","Type":"ContainerStarted","Data":"920494cfbb35ab3cf223c4da98325172b5ea759b577420be6c4fb995b90ac2d1"} Nov 04 04:45:34.677369 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:34.677039 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" event={"ID":"7380461b-403d-4470-8c1c-6433616b1dd2","Type":"ContainerStarted","Data":"30c557f965529fa028161efd235894f1d04aba9782c444fef6830679f7f791fd"} Nov 04 04:45:34.677369 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:34.677066 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" event={"ID":"7380461b-403d-4470-8c1c-6433616b1dd2","Type":"ContainerStarted","Data":"9851dece60add96c4f0939a2e559b14109e74892b263ce56ace0e4aa16d87193"} Nov 04 04:45:34.677369 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:34.677093 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" event={"ID":"7380461b-403d-4470-8c1c-6433616b1dd2","Type":"ContainerStarted","Data":"db78bc9846be7c916d91f56b1a2462a8baa9321a64efc2368b53f43741440ed0"} Nov 04 04:45:35.546552 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:35.546518 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:35.546552 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:35.546533 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfn6w" Nov 04 04:45:35.546773 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:35.546647 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkgd9" Nov 04 04:45:35.546773 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:35.546657 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwgm5" podUID="3e3bf36f-7156-4151-9d6d-0937169dbcca" Nov 04 04:45:35.546773 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:35.546756 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkgd9" podUID="f0c260a1-2a73-4b2b-b6a7-2f9717941a1b" Nov 04 04:45:35.546922 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:35.546849 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfn6w" podUID="59d18119-72a5-410f-be97-90613af651bd" Nov 04 04:45:35.681606 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:35.681578 2578 generic.go:356] "Generic (PLEG): container finished" podID="31e07d9c-93bc-4e8d-909b-e65406ff62df" containerID="bdc7189a4b9c6ef788f0624785d437016f0f231808bec2e065b2727d2a76637c" exitCode=0 Nov 04 04:45:35.681963 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:35.681623 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p9jxj" event={"ID":"31e07d9c-93bc-4e8d-909b-e65406ff62df","Type":"ContainerDied","Data":"bdc7189a4b9c6ef788f0624785d437016f0f231808bec2e065b2727d2a76637c"} Nov 04 04:45:36.687209 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:36.687177 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" event={"ID":"7380461b-403d-4470-8c1c-6433616b1dd2","Type":"ContainerStarted","Data":"f9409cc8a043c80cf851bb83e318f8d22630a1707401549faebe21877bbd9c74"} Nov 04 04:45:36.690097 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:36.690066 2578 generic.go:356] "Generic (PLEG): container finished" podID="31e07d9c-93bc-4e8d-909b-e65406ff62df" containerID="bae6b8fc7b6ca1a06d2c068068f04c091ba31336a9405402b2e9582a5d5f40fc" exitCode=0 Nov 04 04:45:36.690193 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:36.690105 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p9jxj" event={"ID":"31e07d9c-93bc-4e8d-909b-e65406ff62df","Type":"ContainerDied","Data":"bae6b8fc7b6ca1a06d2c068068f04c091ba31336a9405402b2e9582a5d5f40fc"} Nov 04 04:45:37.546700 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:37.546671 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:37.546875 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:37.546766 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfn6w" Nov 04 04:45:37.546875 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:37.546874 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkgd9" Nov 04 04:45:37.546977 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:37.546893 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwgm5" podUID="3e3bf36f-7156-4151-9d6d-0937169dbcca" Nov 04 04:45:37.546977 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:37.546957 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkgd9" podUID="f0c260a1-2a73-4b2b-b6a7-2f9717941a1b" Nov 04 04:45:37.547087 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:37.546995 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfn6w" podUID="59d18119-72a5-410f-be97-90613af651bd" Nov 04 04:45:37.695713 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:37.695680 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p9jxj" event={"ID":"31e07d9c-93bc-4e8d-909b-e65406ff62df","Type":"ContainerStarted","Data":"874f7a518e7fa4d6577c95b08bf1593a4b413e635eea7f0bc3a5a08790d6b1ea"} Nov 04 04:45:37.837983 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:37.837908 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-p9jxj" podStartSLOduration=9.093044496 podStartE2EDuration="21.837896765s" totalImagesPullingTime="12.744852269s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 04:45:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:45:37.837145869 +0000 UTC m=+21.835714743" watchObservedRunningTime="2026-11-04 04:45:37.837896765 +0000 UTC m=+21.836465700" Nov 04 04:45:38.702112 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:38.702080 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" event={"ID":"7380461b-403d-4470-8c1c-6433616b1dd2","Type":"ContainerStarted","Data":"a724f36584b60495a1b515a243cde57c0bef82427033f8701fabba66c054f57b"} Nov 04 04:45:38.939662 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:38.939574 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" podStartSLOduration=8.076779567 podStartE2EDuration="22.93956132s" totalImagesPullingTime="14.862781753s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 04:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:45:38.904798934 +0000 UTC m=+22.903367807" watchObservedRunningTime="2026-11-04 04:45:38.93956132 +0000 UTC m=+22.938130251" Nov 04 04:45:39.546430 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:39.546401 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:39.546588 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:39.546436 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfn6w" Nov 04 04:45:39.546588 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:39.546493 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwgm5" podUID="3e3bf36f-7156-4151-9d6d-0937169dbcca" Nov 04 04:45:39.546588 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:39.546574 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkgd9" Nov 04 04:45:39.546707 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:39.546680 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkgd9" podUID="f0c260a1-2a73-4b2b-b6a7-2f9717941a1b" Nov 04 04:45:39.546767 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:39.546746 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfn6w" podUID="59d18119-72a5-410f-be97-90613af651bd" Nov 04 04:45:39.704582 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:39.704563 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 04:45:39.705021 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:39.705004 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:39.705084 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:39.705073 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:39.720074 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:39.720057 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:39.720331 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:39.720318 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:40.705892 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:40.705863 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 04:45:41.347107 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:41.346877 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mfn6w"] Nov 04 04:45:41.347247 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:41.347160 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfn6w" Nov 04 04:45:41.347285 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:41.347244 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfn6w" podUID="59d18119-72a5-410f-be97-90613af651bd" Nov 04 04:45:41.351122 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:41.351097 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mwgm5"] Nov 04 04:45:41.351240 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:41.351185 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:41.351276 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:41.351254 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwgm5" podUID="3e3bf36f-7156-4151-9d6d-0937169dbcca" Nov 04 04:45:41.404977 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:41.404956 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tkgd9"] Nov 04 04:45:41.405066 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:41.405047 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkgd9" Nov 04 04:45:41.405141 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:41.405124 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkgd9" podUID="f0c260a1-2a73-4b2b-b6a7-2f9717941a1b" Nov 04 04:45:41.707637 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:41.707584 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 04:45:42.547006 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:42.546980 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkgd9" Nov 04 04:45:42.547153 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:42.547081 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkgd9" podUID="f0c260a1-2a73-4b2b-b6a7-2f9717941a1b" Nov 04 04:45:43.546475 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:43.546448 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfn6w" Nov 04 04:45:43.546809 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:43.546449 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:43.546809 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:43.546548 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfn6w" podUID="59d18119-72a5-410f-be97-90613af651bd" Nov 04 04:45:43.546809 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:43.546635 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwgm5" podUID="3e3bf36f-7156-4151-9d6d-0937169dbcca" Nov 04 04:45:44.546594 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:44.546565 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkgd9" Nov 04 04:45:44.547158 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:44.546678 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkgd9" podUID="f0c260a1-2a73-4b2b-b6a7-2f9717941a1b" Nov 04 04:45:45.546872 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:45.546849 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:45.547241 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:45.546880 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfn6w" Nov 04 04:45:45.547241 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:45.546945 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwgm5" podUID="3e3bf36f-7156-4151-9d6d-0937169dbcca" Nov 04 04:45:45.547241 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:45.546999 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfn6w" podUID="59d18119-72a5-410f-be97-90613af651bd" Nov 04 04:45:46.547194 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:46.547165 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkgd9" Nov 04 04:45:46.547596 ip-10-0-134-224 kubenswrapper[2578]: E1104 04:45:46.547247 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tkgd9" podUID="f0c260a1-2a73-4b2b-b6a7-2f9717941a1b" Nov 04 04:45:47.013465 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:47.013438 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:47.013631 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:47.013619 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 04:45:47.030512 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:47.030492 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:47.278910 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:47.278847 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeReady" Nov 04 04:45:47.279016 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:47.278932 2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 04 04:45:47.546961 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:47.546905 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:47.547085 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:47.547075 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfn6w" Nov 04 04:45:47.609081 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:47.609056 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 04 04:45:47.609081 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:47.609074 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 04 04:45:47.609481 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:47.609266 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rrdk2\"" Nov 04 04:45:47.609481 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:47.609421 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 04 04:45:47.609481 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:47.609436 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-45k46\"" Nov 04 04:45:48.353406 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:48.353377 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7f22t_52f02c25-48ed-46dc-82bc-23016ffa595d/dns-node-resolver/0.log" Nov 04 04:45:48.546814 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:48.546786 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkgd9" Nov 04 04:45:48.607375 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:48.607326 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-m97m8\"" Nov 04 04:45:48.607375 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:48.607339 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 04 04:45:48.923041 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:48.922993 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7tlqm_5c1153d6-83db-4eb7-95e0-78d09bd61d6d/node-ca/0.log" Nov 04 04:45:49.469476 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:49.469449 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3e3bf36f-7156-4151-9d6d-0937169dbcca-original-pull-secret\") pod \"global-pull-secret-syncer-mwgm5\" (UID: \"3e3bf36f-7156-4151-9d6d-0937169dbcca\") " pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:49.469586 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:49.469495 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0c260a1-2a73-4b2b-b6a7-2f9717941a1b-metrics-certs\") pod \"network-metrics-daemon-tkgd9\" (UID: \"f0c260a1-2a73-4b2b-b6a7-2f9717941a1b\") " pod="openshift-multus/network-metrics-daemon-tkgd9" Nov 04 04:45:49.473713 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:49.473679 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0c260a1-2a73-4b2b-b6a7-2f9717941a1b-metrics-certs\") pod \"network-metrics-daemon-tkgd9\" (UID: \"f0c260a1-2a73-4b2b-b6a7-2f9717941a1b\") " pod="openshift-multus/network-metrics-daemon-tkgd9" Nov 04 04:45:49.473840 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:49.473803 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3e3bf36f-7156-4151-9d6d-0937169dbcca-original-pull-secret\") pod \"global-pull-secret-syncer-mwgm5\" (UID: \"3e3bf36f-7156-4151-9d6d-0937169dbcca\") " pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:49.491154 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:49.491131 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tkgd9" Nov 04 04:45:49.569963 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:49.569936 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7jdp\" (UniqueName: \"kubernetes.io/projected/59d18119-72a5-410f-be97-90613af651bd-kube-api-access-b7jdp\") pod \"network-check-target-mfn6w\" (UID: \"59d18119-72a5-410f-be97-90613af651bd\") " pod="openshift-network-diagnostics/network-check-target-mfn6w" Nov 04 04:45:49.572540 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:49.572514 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7jdp\" (UniqueName: \"kubernetes.io/projected/59d18119-72a5-410f-be97-90613af651bd-kube-api-access-b7jdp\") pod \"network-check-target-mfn6w\" (UID: \"59d18119-72a5-410f-be97-90613af651bd\") " pod="openshift-network-diagnostics/network-check-target-mfn6w" Nov 04 04:45:49.680187 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:49.680161 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:49.693412 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:49.693391 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfn6w" Nov 04 04:45:49.921016 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:49.920975 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0c260a1_2a73_4b2b_b6a7_2f9717941a1b.slice/crio-5a32a8e915e44453cceb3a6c6b4f03f2445cffc4f9c880e999a1c353ab64db22 WatchSource:0}: Error finding container 5a32a8e915e44453cceb3a6c6b4f03f2445cffc4f9c880e999a1c353ab64db22: Status 404 returned error can't find the container with id 5a32a8e915e44453cceb3a6c6b4f03f2445cffc4f9c880e999a1c353ab64db22 Nov 04 04:45:49.923699 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:49.923682 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tkgd9"] Nov 04 04:45:50.045453 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:50.045430 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e3bf36f_7156_4151_9d6d_0937169dbcca.slice/crio-7ccb2c5cf6cf822ca5d1c4c7b51f5f08960eb61b3971ebad33d057addc0bf835 WatchSource:0}: Error finding container 7ccb2c5cf6cf822ca5d1c4c7b51f5f08960eb61b3971ebad33d057addc0bf835: Status 404 returned error can't find the container with id 7ccb2c5cf6cf822ca5d1c4c7b51f5f08960eb61b3971ebad33d057addc0bf835 Nov 04 04:45:50.048736 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:50.048715 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mwgm5"] Nov 04 04:45:50.065530 ip-10-0-134-224 kubenswrapper[2578]: W1104 04:45:50.065511 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59d18119_72a5_410f_be97_90613af651bd.slice/crio-39dafa117e7beae7f40ed505d989b9fe9b6b2ae29b88f4da370722826eaad31e WatchSource:0}: Error finding container 39dafa117e7beae7f40ed505d989b9fe9b6b2ae29b88f4da370722826eaad31e: Status 404 returned error can't find the container with id 39dafa117e7beae7f40ed505d989b9fe9b6b2ae29b88f4da370722826eaad31e Nov 04 04:45:50.068976 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:50.068954 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mfn6w"] Nov 04 04:45:50.726105 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:50.726074 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mfn6w" event={"ID":"59d18119-72a5-410f-be97-90613af651bd","Type":"ContainerStarted","Data":"39dafa117e7beae7f40ed505d989b9fe9b6b2ae29b88f4da370722826eaad31e"} Nov 04 04:45:50.727324 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:50.727282 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mwgm5" event={"ID":"3e3bf36f-7156-4151-9d6d-0937169dbcca","Type":"ContainerStarted","Data":"7ccb2c5cf6cf822ca5d1c4c7b51f5f08960eb61b3971ebad33d057addc0bf835"} Nov 04 04:45:50.728487 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:50.728464 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tkgd9" event={"ID":"f0c260a1-2a73-4b2b-b6a7-2f9717941a1b","Type":"ContainerStarted","Data":"5a32a8e915e44453cceb3a6c6b4f03f2445cffc4f9c880e999a1c353ab64db22"} Nov 04 04:45:54.740708 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:54.740670 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tkgd9" event={"ID":"f0c260a1-2a73-4b2b-b6a7-2f9717941a1b","Type":"ContainerStarted","Data":"ad229b54e5153d11aea2aba349ef85854795862c21ac42aaacdccc6d22172dc4"} Nov 04 04:45:54.741131 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:54.740736 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tkgd9" event={"ID":"f0c260a1-2a73-4b2b-b6a7-2f9717941a1b","Type":"ContainerStarted","Data":"0ee32620ed54c7db6fc93e7a6847db3d2a6298198f1b444ccc6c453d74fc7df3"} Nov 04 04:45:54.872418 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:54.872367 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tkgd9" podStartSLOduration=35.110248625 podStartE2EDuration="38.872349033s" totalImagesPullingTime="3.762100408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:45:54.871665877 +0000 UTC m=+38.870234752" watchObservedRunningTime="2026-11-04 04:45:54.872349033 +0000 UTC m=+38.870917907" Nov 04 04:45:56.746616 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:56.746578 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mfn6w" event={"ID":"59d18119-72a5-410f-be97-90613af651bd","Type":"ContainerStarted","Data":"b8a36b8d7a1cfd5f521cb33ffb8f3e4adf7dc33c66577e14f3873f0eeb686aec"} Nov 04 04:45:56.747054 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:56.746716 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mfn6w" Nov 04 04:45:56.877637 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:56.877581 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mfn6w" podStartSLOduration=34.859568052 podStartE2EDuration="40.877564008s" totalImagesPullingTime="6.017995956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:45:56.876819756 +0000 UTC m=+40.875388631" watchObservedRunningTime="2026-11-04 04:45:56.877564008 +0000 UTC m=+40.876132881" Nov 04 04:45:58.263647 ip-10-0-134-224 kubenswrapper[2578]: I1104 04:45:58.263561 2578 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 04:45:58.263750 ip-10-0-134-224 systemd[1]: Stopping Kubernetes Kubelet... Nov 04 04:45:58.272814 ip-10-0-134-224 systemd[1]: kubelet.service: Deactivated successfully. Nov 04 04:45:58.272976 ip-10-0-134-224 systemd[1]: Stopped Kubernetes Kubelet. Nov 04 04:45:58.273022 ip-10-0-134-224 systemd[1]: kubelet.service: Consumed 2.127s CPU time. Nov 04 04:45:58.280866 ip-10-0-134-224 systemd[1]: Starting Kubernetes Kubelet... Nov 04 04:45:58.341050 ip-10-0-134-224 kubenswrapper[4395]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 04:45:58.341050 ip-10-0-134-224 kubenswrapper[4395]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 04 04:45:58.341050 ip-10-0-134-224 kubenswrapper[4395]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 04:45:58.341050 ip-10-0-134-224 kubenswrapper[4395]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 04:45:58.343200 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343134 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:45:58.343200 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343201 4395 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:45:58.343271 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343205 4395 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:45:58.343271 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343208 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:45:58.343271 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343211 4395 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:45:58.343271 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343215 4395 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:45:58.343271 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343218 4395 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:45:58.343271 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343224 4395 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:45:58.343271 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343228 4395 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:45:58.343271 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343231 4395 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:45:58.343271 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343234 4395 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:45:58.343271 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343237 4395 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:45:58.343271 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343239 4395 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:45:58.343271 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343242 4395 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:45:58.343271 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343245 4395 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:45:58.343271 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343247 4395 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:45:58.343271 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343250 4395 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:45:58.343271 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343253 4395 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:45:58.343271 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343255 4395 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:45:58.343271 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343259 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:45:58.343271 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343262 4395 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:45:58.343761 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343265 4395 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:45:58.343761 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343267 4395 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:45:58.343761 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343270 4395 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:45:58.343761 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343273 4395 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:45:58.343761 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343276 4395 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:45:58.343761 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343279 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:45:58.343761 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343282 4395 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:45:58.343761 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343285 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:45:58.343761 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343287 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:45:58.343761 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343290 4395 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:45:58.343761 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343308 4395 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:45:58.343761 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343313 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:45:58.343761 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343316 4395 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:45:58.343761 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343318 4395 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:45:58.343761 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343321 4395 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:45:58.343761 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343323 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:45:58.343761 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343326 4395 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:45:58.343761 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343329 4395 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:45:58.343761 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343333 4395 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:45:58.344268 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343335 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:45:58.344268 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343338 4395 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:45:58.344268 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343341 4395 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:45:58.344268 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343344 4395 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:45:58.344268 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343347 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:45:58.344268 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343350 4395 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:45:58.344268 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343353 4395 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:45:58.344268 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343356 4395 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:45:58.344268 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343358 4395 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:45:58.344268 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343361 4395 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:45:58.344268 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343364 4395 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:45:58.344268 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343366 4395 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:45:58.344268 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343369 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:45:58.344268 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343372 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:45:58.344268 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343375 4395 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:45:58.344268 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343377 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:45:58.344268 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343380 4395 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:45:58.344268 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343383 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:45:58.344268 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343386 4395 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:45:58.344268 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343388 4395 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:45:58.344795 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343391 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:45:58.344795 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343393 4395 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:45:58.344795 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343396 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:45:58.344795 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343398 4395 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:45:58.344795 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343401 4395 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:45:58.344795 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343404 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:45:58.344795 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343407 4395 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:45:58.344795 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343410 4395 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:45:58.344795 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343412 4395 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:45:58.344795 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343415 4395 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:45:58.344795 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343417 4395 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:45:58.344795 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343420 4395 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:45:58.344795 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343423 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:45:58.344795 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343426 4395 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:45:58.344795 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343429 4395 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:45:58.344795 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343433 4395 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:45:58.344795 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343438 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:45:58.344795 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343441 4395 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:45:58.344795 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343444 4395 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:45:58.345262 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343447 4395 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:45:58.345262 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343449 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:45:58.345262 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343452 4395 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:45:58.345262 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343455 4395 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:45:58.345262 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343458 4395 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:45:58.345262 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343460 4395 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:45:58.345262 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343463 4395 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:45:58.345262 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343466 4395 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:45:58.345262 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343468 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:45:58.345262 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343471 4395 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:45:58.345262 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343473 4395 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:45:58.345262 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343476 4395 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:45:58.345262 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343479 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:45:58.345262 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343482 4395 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:45:58.345262 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343486 4395 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:45:58.345262 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343490 4395 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:45:58.345262 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343493 4395 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:45:58.345262 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343496 4395 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:45:58.345262 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343499 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:45:58.345262 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343501 4395 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:45:58.345774 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343504 4395 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:45:58.345774 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343506 4395 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:45:58.345774 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343509 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:45:58.345774 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343513 4395 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:45:58.345774 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343516 4395 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:45:58.345774 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343519 4395 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:45:58.345774 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.343522 4395 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:45:58.345774 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344389 4395 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:45:58.345774 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344398 4395 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:45:58.345774 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344401 4395 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:45:58.345774 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344404 4395 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:45:58.345774 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344408 4395 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:45:58.345774 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344411 4395 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:45:58.345774 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344414 4395 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:45:58.345774 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344417 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:45:58.345774 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344419 4395 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:45:58.345774 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344422 4395 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:45:58.345774 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344425 4395 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:45:58.345774 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344427 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:45:58.346278 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344430 4395 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:45:58.346278 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344432 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:45:58.346278 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344435 4395 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:45:58.346278 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344438 4395 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:45:58.346278 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344440 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:45:58.346278 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344443 4395 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:45:58.346278 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344446 4395 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:45:58.346278 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344449 4395 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:45:58.346278 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344452 4395 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:45:58.346278 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344454 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:45:58.346278 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344457 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:45:58.346278 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344460 4395 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:45:58.346278 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344463 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:45:58.346278 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344465 4395 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:45:58.346278 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344468 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:45:58.346278 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344470 4395 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:45:58.346278 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344473 4395 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:45:58.346278 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344476 4395 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:45:58.346278 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344478 4395 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:45:58.346770 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344481 4395 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:45:58.346770 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344489 4395 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:45:58.346770 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344492 4395 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:45:58.346770 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344495 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:45:58.346770 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344497 4395 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:45:58.346770 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344500 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:45:58.346770 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344504 4395 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:45:58.346770 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344508 4395 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:45:58.346770 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344511 4395 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:45:58.346770 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344514 4395 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:45:58.346770 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344517 4395 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:45:58.346770 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344519 4395 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:45:58.346770 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344522 4395 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:45:58.346770 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344525 4395 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:45:58.346770 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344527 4395 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:45:58.346770 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344530 4395 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:45:58.346770 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344532 4395 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:45:58.346770 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344535 4395 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:45:58.346770 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344537 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:45:58.346770 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344540 4395 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:45:58.347263 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344542 4395 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:45:58.347263 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344545 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:45:58.347263 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344547 4395 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:45:58.347263 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344552 4395 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:45:58.347263 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344555 4395 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:45:58.347263 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344558 4395 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:45:58.347263 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344561 4395 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:45:58.347263 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344563 4395 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:45:58.347263 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344565 4395 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:45:58.347263 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344568 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:45:58.347263 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344571 4395 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:45:58.347263 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344575 4395 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:45:58.347263 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344579 4395 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:45:58.347263 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344583 4395 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:45:58.347263 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344591 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:45:58.347263 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344594 4395 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:45:58.347263 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344597 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:45:58.347263 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344599 4395 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:45:58.347263 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344602 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:45:58.347770 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344604 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:45:58.347770 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344607 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:45:58.347770 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344610 4395 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:45:58.347770 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344613 4395 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:45:58.347770 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344616 4395 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:45:58.347770 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344618 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:45:58.347770 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344621 4395 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:45:58.347770 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344624 4395 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:45:58.347770 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344627 4395 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:45:58.347770 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344629 4395 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:45:58.347770 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344632 4395 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:45:58.347770 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344634 4395 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:45:58.347770 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344637 4395 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:45:58.347770 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344639 4395 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:45:58.347770 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344642 4395 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:45:58.347770 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344644 4395 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:45:58.347770 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344649 4395 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:45:58.347770 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344652 4395 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:45:58.347770 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344654 4395 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:45:58.348231 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344657 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:45:58.348231 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344660 4395 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:45:58.348231 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344662 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:45:58.348231 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344665 4395 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:45:58.348231 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344667 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:45:58.348231 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344676 4395 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:45:58.348231 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344679 4395 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:45:58.348231 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344681 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:45:58.348231 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344684 4395 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:45:58.348231 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344692 4395 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:45:58.348231 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344695 4395 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:45:58.348231 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344697 4395 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:45:58.348231 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344700 4395 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:45:58.348231 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344702 4395 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:45:58.348231 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344705 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:45:58.348231 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344709 4395 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:45:58.348231 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.344711 4395 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:45:58.348231 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345259 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:45:58.348231 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345264 4395 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:45:58.348231 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345269 4395 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:45:58.348732 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345274 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:45:58.348732 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345277 4395 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:45:58.348732 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345280 4395 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:45:58.348732 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345284 4395 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:45:58.348732 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345287 4395 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:45:58.348732 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345290 4395 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:45:58.348732 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345292 4395 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:45:58.348732 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345311 4395 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:45:58.348732 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345316 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:45:58.348732 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345325 4395 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:45:58.348732 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345329 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:45:58.348732 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345332 4395 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:45:58.348732 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345335 4395 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:45:58.348732 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345339 4395 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:45:58.348732 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345342 4395 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:45:58.348732 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345345 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:45:58.348732 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345347 4395 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:45:58.348732 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345350 4395 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:45:58.348732 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345354 4395 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:45:58.349214 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345357 4395 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:45:58.349214 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345360 4395 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:45:58.349214 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345370 4395 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:45:58.349214 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345373 4395 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:45:58.349214 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345375 4395 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:45:58.349214 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345378 4395 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:45:58.349214 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345381 4395 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:45:58.349214 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345383 4395 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:45:58.349214 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345387 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:45:58.349214 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345390 4395 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:45:58.349214 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345392 4395 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:45:58.349214 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345395 4395 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:45:58.349214 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345398 4395 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:45:58.349214 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345401 4395 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:45:58.349214 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345404 4395 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:45:58.349214 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345406 4395 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:45:58.349214 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345409 4395 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:45:58.349214 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345411 4395 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:45:58.349214 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345414 4395 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:45:58.349214 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345416 4395 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:45:58.349722 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345419 4395 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:45:58.349722 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345422 4395 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:45:58.349722 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345427 4395 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:45:58.349722 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345430 4395 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:45:58.349722 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345432 4395 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:45:58.349722 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345435 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:45:58.349722 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345437 4395 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:45:58.349722 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345440 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:45:58.349722 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345442 4395 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:45:58.349722 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345444 4395 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:45:58.349722 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345447 4395 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:45:58.349722 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345449 4395 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:45:58.349722 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345452 4395 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:45:58.349722 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345454 4395 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:45:58.349722 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345457 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:45:58.349722 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345465 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:45:58.349722 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345468 4395 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:45:58.349722 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345471 4395 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:45:58.349722 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345473 4395 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:45:58.349722 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345476 4395 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:45:58.350210 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345482 4395 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:45:58.350210 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345484 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:45:58.350210 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345487 4395 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:45:58.350210 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345490 4395 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:45:58.350210 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345492 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:45:58.350210 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345495 4395 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:45:58.350210 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345498 4395 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:45:58.350210 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345500 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:45:58.350210 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345503 4395 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:45:58.350210 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345506 4395 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:45:58.350210 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345508 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:45:58.350210 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345511 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:45:58.350210 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345513 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:45:58.350210 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345516 4395 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:45:58.350210 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345520 4395 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:45:58.350210 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345523 4395 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:45:58.350210 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345525 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:45:58.350210 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345528 4395 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:45:58.350210 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345530 4395 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:45:58.350693 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345533 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:45:58.350693 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345536 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:45:58.350693 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345538 4395 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:45:58.350693 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345541 4395 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:45:58.350693 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345543 4395 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:45:58.350693 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345546 4395 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:45:58.350693 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345548 4395 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:45:58.350693 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345552 4395 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:45:58.350693 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345554 4395 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:45:58.350693 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345558 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:45:58.350693 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345560 4395 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:45:58.350693 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345563 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:45:58.350693 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345566 4395 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:45:58.350693 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345569 4395 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:45:58.350693 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345572 4395 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:45:58.350693 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345575 4395 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:45:58.350693 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345577 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:45:58.350693 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345580 4395 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:45:58.350693 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345583 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:45:58.351154 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345585 4395 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:45:58.351154 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345588 4395 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:45:58.351154 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345590 4395 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:45:58.351154 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345593 4395 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:45:58.351154 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345596 4395 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:45:58.351154 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.345598 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:45:58.351154 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345690 4395 flags.go:64] FLAG: --address="0.0.0.0" Nov 04 04:45:58.351154 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345697 4395 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 04 04:45:58.351154 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345712 4395 flags.go:64] FLAG: --anonymous-auth="true" Nov 04 04:45:58.351154 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345717 4395 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 04 04:45:58.351154 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345721 4395 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 04 04:45:58.351154 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345725 4395 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 04 04:45:58.351154 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345729 4395 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 04 04:45:58.351154 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345733 4395 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 04 04:45:58.351154 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345737 4395 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 04 04:45:58.351154 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345740 4395 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 04 04:45:58.351154 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345743 4395 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 04 04:45:58.351154 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345747 4395 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 04 04:45:58.351154 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345750 4395 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 04 04:45:58.351154 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345753 4395 flags.go:64] FLAG: --cgroup-root="" Nov 04 04:45:58.351154 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345756 4395 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 04 04:45:58.351154 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345760 4395 flags.go:64] FLAG: --client-ca-file="" Nov 04 04:45:58.351154 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345763 4395 flags.go:64] FLAG: --cloud-provider="external" Nov 04 04:45:58.351766 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345766 4395 flags.go:64] FLAG: --cluster-dns="[]" Nov 04 04:45:58.351766 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345770 4395 flags.go:64] FLAG: --cluster-domain="" Nov 04 04:45:58.351766 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345773 4395 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 04 04:45:58.351766 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345778 4395 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 04 04:45:58.351766 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345781 4395 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 04 04:45:58.351766 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345785 4395 flags.go:64] FLAG: --container-log-max-files="5" Nov 04 04:45:58.351766 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345789 4395 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 04 04:45:58.351766 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345792 4395 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 04 04:45:58.351766 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345795 4395 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 04 04:45:58.351766 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345799 4395 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 04 04:45:58.351766 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345802 4395 flags.go:64] FLAG: --contention-profiling="false" Nov 04 04:45:58.351766 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345805 4395 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 04 04:45:58.351766 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345808 4395 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 04 04:45:58.351766 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345811 4395 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 04 04:45:58.351766 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345814 4395 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 04 04:45:58.351766 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345818 4395 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 04 04:45:58.351766 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345822 4395 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 04 04:45:58.351766 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345826 4395 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 04 04:45:58.351766 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345828 4395 flags.go:64] FLAG: --enable-load-reader="false" Nov 04 04:45:58.351766 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345831 4395 flags.go:64] FLAG: --enable-server="true" Nov 04 04:45:58.351766 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345835 4395 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 04 04:45:58.351766 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345838 4395 flags.go:64] FLAG: --event-burst="100" Nov 04 04:45:58.351766 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345842 4395 flags.go:64] FLAG: --event-qps="50" Nov 04 04:45:58.351766 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345845 4395 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 04 04:45:58.352495 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345848 4395 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 04 04:45:58.352495 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345851 4395 flags.go:64] FLAG: --eviction-hard="" Nov 04 04:45:58.352495 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345855 4395 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 04 04:45:58.352495 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345858 4395 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 04 04:45:58.352495 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345861 4395 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 04 04:45:58.352495 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345864 4395 flags.go:64] FLAG: --eviction-soft="" Nov 04 04:45:58.352495 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345867 4395 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 04 04:45:58.352495 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345870 4395 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 04 04:45:58.352495 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345873 4395 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 04 04:45:58.352495 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345877 4395 flags.go:64] FLAG: --experimental-mounter-path="" Nov 04 04:45:58.352495 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345880 4395 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 04 04:45:58.352495 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345884 4395 flags.go:64] FLAG: --fail-swap-on="true" Nov 04 04:45:58.352495 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345888 4395 flags.go:64] FLAG: --feature-gates="" Nov 04 04:45:58.352495 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345892 4395 flags.go:64] FLAG: --file-check-frequency="20s" Nov 04 04:45:58.352495 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345895 4395 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 04 04:45:58.352495 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345898 4395 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 04 04:45:58.352495 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345901 4395 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 04 04:45:58.352495 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345904 4395 flags.go:64] FLAG: --healthz-port="10248" Nov 04 04:45:58.352495 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345907 4395 flags.go:64] FLAG: --help="false" Nov 04 04:45:58.352495 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345911 4395 flags.go:64] FLAG: --hostname-override="ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:58.352495 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345914 4395 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 04 04:45:58.352495 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345917 4395 flags.go:64] FLAG: --http-check-frequency="20s" Nov 04 04:45:58.352495 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345920 4395 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 04 04:45:58.353346 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345923 4395 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 04 04:45:58.353346 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345927 4395 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 04 04:45:58.353346 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345930 4395 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 04 04:45:58.353346 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345933 4395 flags.go:64] FLAG: --image-service-endpoint="" Nov 04 04:45:58.353346 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345936 4395 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 04 04:45:58.353346 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345939 4395 flags.go:64] FLAG: --kube-api-burst="100" Nov 04 04:45:58.353346 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345942 4395 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 04 04:45:58.353346 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345945 4395 flags.go:64] FLAG: --kube-api-qps="50" Nov 04 04:45:58.353346 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345948 4395 flags.go:64] FLAG: --kube-reserved="" Nov 04 04:45:58.353346 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345951 4395 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 04 04:45:58.353346 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345954 4395 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 04 04:45:58.353346 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345957 4395 flags.go:64] FLAG: --kubelet-cgroups="" Nov 04 04:45:58.353346 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345960 4395 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 04 04:45:58.353346 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345963 4395 flags.go:64] FLAG: --lock-file="" Nov 04 04:45:58.353346 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345966 4395 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 04 04:45:58.353346 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345969 4395 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 04 04:45:58.353346 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345972 4395 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 04 04:45:58.353346 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345977 4395 flags.go:64] FLAG: --log-json-split-stream="false" Nov 04 04:45:58.353346 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345980 4395 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 04 04:45:58.353346 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345984 4395 flags.go:64] FLAG: --log-text-split-stream="false" Nov 04 04:45:58.353346 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345988 4395 flags.go:64] FLAG: --logging-format="text" Nov 04 04:45:58.353346 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345991 4395 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 04 04:45:58.353346 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345995 4395 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 04 04:45:58.353346 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.345998 4395 flags.go:64] FLAG: --manifest-url="" Nov 04 04:45:58.353953 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346000 4395 flags.go:64] FLAG: --manifest-url-header="" Nov 04 04:45:58.353953 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346005 4395 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 04 04:45:58.353953 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346008 4395 flags.go:64] FLAG: --max-open-files="1000000" Nov 04 04:45:58.353953 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346012 4395 flags.go:64] FLAG: --max-pods="110" Nov 04 04:45:58.353953 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346015 4395 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 04 04:45:58.353953 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346018 4395 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 04 04:45:58.353953 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346021 4395 flags.go:64] FLAG: --memory-manager-policy="None" Nov 04 04:45:58.353953 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346024 4395 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 04 04:45:58.353953 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346027 4395 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 04 04:45:58.353953 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346030 4395 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 04 04:45:58.353953 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346033 4395 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 04 04:45:58.353953 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346040 4395 flags.go:64] FLAG: --node-status-max-images="50" Nov 04 04:45:58.353953 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346044 4395 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 04 04:45:58.353953 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346047 4395 flags.go:64] FLAG: --oom-score-adj="-999" Nov 04 04:45:58.353953 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346050 4395 flags.go:64] FLAG: --pod-cidr="" Nov 04 04:45:58.353953 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346053 4395 flags.go:64] FLAG: --pod-manifest-path="" Nov 04 04:45:58.353953 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346056 4395 flags.go:64] FLAG: --pod-max-pids="-1" Nov 04 04:45:58.353953 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346059 4395 flags.go:64] FLAG: --pods-per-core="0" Nov 04 04:45:58.353953 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346062 4395 flags.go:64] FLAG: --port="10250" Nov 04 04:45:58.353953 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346065 4395 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 04 04:45:58.353953 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346068 4395 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0c85fe216f8cd8da7" Nov 04 04:45:58.353953 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346071 4395 flags.go:64] FLAG: --qos-reserved="" Nov 04 04:45:58.353953 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346074 4395 flags.go:64] FLAG: --read-only-port="10255" Nov 04 04:45:58.353953 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346077 4395 flags.go:64] FLAG: --register-node="true" Nov 04 04:45:58.353953 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346080 4395 flags.go:64] FLAG: --register-with-taints="" Nov 04 04:45:58.353953 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346084 4395 flags.go:64] FLAG: --registry-burst="10" Nov 04 04:45:58.354586 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346088 4395 flags.go:64] FLAG: --registry-qps="5" Nov 04 04:45:58.354586 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346091 4395 flags.go:64] FLAG: --reserved-cpus="" Nov 04 04:45:58.354586 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346095 4395 flags.go:64] FLAG: --reserved-memory="" Nov 04 04:45:58.354586 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346099 4395 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 04 04:45:58.354586 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346102 4395 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 04 04:45:58.354586 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346106 4395 flags.go:64] FLAG: --rotate-certificates="false" Nov 04 04:45:58.354586 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346109 4395 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 04 04:45:58.354586 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346112 4395 flags.go:64] FLAG: --runonce="false" Nov 04 04:45:58.354586 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346115 4395 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 04 04:45:58.354586 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346118 4395 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 04 04:45:58.354586 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346121 4395 flags.go:64] FLAG: --seccomp-default="false" Nov 04 04:45:58.354586 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346124 4395 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 04 04:45:58.354586 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346127 4395 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 04 04:45:58.354586 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346130 4395 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 04 04:45:58.354586 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346133 4395 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 04 04:45:58.354586 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346136 4395 flags.go:64] FLAG: --storage-driver-password="root" Nov 04 04:45:58.354586 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346139 4395 flags.go:64] FLAG: --storage-driver-secure="false" Nov 04 04:45:58.354586 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346142 4395 flags.go:64] FLAG: --storage-driver-table="stats" Nov 04 04:45:58.354586 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346144 4395 flags.go:64] FLAG: --storage-driver-user="root" Nov 04 04:45:58.354586 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346147 4395 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 04 04:45:58.354586 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346150 4395 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 04 04:45:58.354586 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346153 4395 flags.go:64] FLAG: --system-cgroups="" Nov 04 04:45:58.354586 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346156 4395 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 04 04:45:58.354586 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346161 4395 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 04 04:45:58.354586 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346164 4395 flags.go:64] FLAG: --tls-cert-file="" Nov 04 04:45:58.355258 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346167 4395 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 04 04:45:58.355258 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346171 4395 flags.go:64] FLAG: --tls-min-version="" Nov 04 04:45:58.355258 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346175 4395 flags.go:64] FLAG: --tls-private-key-file="" Nov 04 04:45:58.355258 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346178 4395 flags.go:64] FLAG: --topology-manager-policy="none" Nov 04 04:45:58.355258 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346181 4395 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 04 04:45:58.355258 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346184 4395 flags.go:64] FLAG: --topology-manager-scope="container" Nov 04 04:45:58.355258 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346187 4395 flags.go:64] FLAG: --v="2" Nov 04 04:45:58.355258 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346191 4395 flags.go:64] FLAG: --version="false" Nov 04 04:45:58.355258 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346195 4395 flags.go:64] FLAG: --vmodule="" Nov 04 04:45:58.355258 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346200 4395 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 04 04:45:58.355258 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346205 4395 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 04 04:45:58.355258 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346386 4395 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:45:58.355258 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346390 4395 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:45:58.355258 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346394 4395 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:45:58.355258 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346397 4395 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:45:58.355258 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346399 4395 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:45:58.355258 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346405 4395 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:45:58.355258 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346408 4395 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:45:58.355258 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346410 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:45:58.355258 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346413 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:45:58.355258 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346416 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:45:58.355258 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346418 4395 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:45:58.355258 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346421 4395 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:45:58.355827 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346424 4395 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:45:58.355827 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346426 4395 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:45:58.355827 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346429 4395 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:45:58.355827 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346431 4395 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:45:58.355827 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346434 4395 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:45:58.355827 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346436 4395 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:45:58.355827 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346439 4395 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:45:58.355827 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346441 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:45:58.355827 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346444 4395 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:45:58.355827 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346446 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:45:58.355827 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346449 4395 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:45:58.355827 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346453 4395 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:45:58.355827 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346455 4395 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:45:58.355827 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346458 4395 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:45:58.355827 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346461 4395 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:45:58.355827 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346465 4395 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:45:58.355827 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346469 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:45:58.355827 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346472 4395 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:45:58.355827 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346476 4395 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:45:58.355827 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346479 4395 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:45:58.356345 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346482 4395 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:45:58.356345 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346485 4395 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:45:58.356345 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346488 4395 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:45:58.356345 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346491 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:45:58.356345 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346495 4395 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:45:58.356345 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346499 4395 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:45:58.356345 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346502 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:45:58.356345 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346505 4395 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:45:58.356345 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346508 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:45:58.356345 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346510 4395 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:45:58.356345 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346513 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:45:58.356345 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346516 4395 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:45:58.356345 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346518 4395 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:45:58.356345 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346521 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:45:58.356345 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346524 4395 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:45:58.356345 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346526 4395 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:45:58.356345 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346529 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:45:58.356345 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346531 4395 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:45:58.356345 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346534 4395 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:45:58.356861 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346537 4395 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:45:58.356861 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346539 4395 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:45:58.356861 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346542 4395 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:45:58.356861 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346544 4395 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:45:58.356861 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346548 4395 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:45:58.356861 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346551 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:45:58.356861 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346553 4395 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:45:58.356861 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346556 4395 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:45:58.356861 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346559 4395 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:45:58.356861 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346561 4395 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:45:58.356861 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346564 4395 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:45:58.356861 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346567 4395 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:45:58.356861 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346570 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:45:58.356861 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346573 4395 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:45:58.356861 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346576 4395 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:45:58.356861 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346579 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:45:58.356861 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346581 4395 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:45:58.356861 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346584 4395 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:45:58.356861 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346587 4395 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:45:58.357428 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346589 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:45:58.357428 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346592 4395 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:45:58.357428 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346594 4395 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:45:58.357428 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346597 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:45:58.357428 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346600 4395 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:45:58.357428 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346602 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:45:58.357428 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346605 4395 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:45:58.357428 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346607 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:45:58.357428 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346610 4395 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:45:58.357428 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346612 4395 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:45:58.357428 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346615 4395 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:45:58.357428 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346618 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:45:58.357428 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346620 4395 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:45:58.357428 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346623 4395 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:45:58.357428 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346625 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:45:58.357428 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346628 4395 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:45:58.357428 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346631 4395 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:45:58.357428 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346634 4395 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:45:58.357428 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346637 4395 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:45:58.357428 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346640 4395 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:45:58.358096 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346643 4395 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:45:58.358096 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346645 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:45:58.358096 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346648 4395 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:45:58.358096 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346650 4395 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:45:58.358096 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346654 4395 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:45:58.358096 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346657 4395 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:45:58.358096 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346659 4395 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:45:58.358096 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346662 4395 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:45:58.358096 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346665 4395 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:45:58.358096 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346668 4395 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:45:58.358096 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346671 4395 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:45:58.358096 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346673 4395 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:45:58.358096 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346676 4395 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:45:58.358096 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346679 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:45:58.358096 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346681 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:45:58.358096 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346684 4395 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:45:58.358546 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.346710 4395 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 04:45:58.358546 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346747 4395 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:45:58.358546 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346751 4395 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:45:58.358546 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346754 4395 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:45:58.358546 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346757 4395 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:45:58.358546 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346760 4395 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:45:58.358546 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346763 4395 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:45:58.358546 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346765 4395 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:45:58.358546 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346768 4395 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:45:58.358546 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346771 4395 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:45:58.358546 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346774 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:45:58.358546 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346777 4395 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:45:58.358546 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346779 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:45:58.358546 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346784 4395 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:45:58.358546 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346787 4395 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:45:58.358546 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346789 4395 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:45:58.359047 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346792 4395 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:45:58.359047 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346795 4395 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:45:58.359047 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346799 4395 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:45:58.359047 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346803 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:45:58.359047 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346806 4395 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:45:58.359047 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346810 4395 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:45:58.359047 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346813 4395 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:45:58.359047 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346815 4395 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:45:58.359047 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346818 4395 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:45:58.359047 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346821 4395 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:45:58.359047 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346824 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:45:58.359047 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346828 4395 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:45:58.359047 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346831 4395 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:45:58.359047 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346834 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:45:58.359047 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346837 4395 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:45:58.359047 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346839 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:45:58.359047 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346842 4395 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:45:58.359047 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346844 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:45:58.359047 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346847 4395 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:45:58.359745 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346849 4395 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:45:58.359745 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346852 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:45:58.359745 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346854 4395 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:45:58.359745 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346857 4395 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:45:58.359745 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346859 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:45:58.359745 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346862 4395 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:45:58.359745 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346865 4395 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:45:58.359745 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346867 4395 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:45:58.359745 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346870 4395 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:45:58.359745 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346873 4395 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:45:58.359745 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346877 4395 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:45:58.359745 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346880 4395 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:45:58.359745 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346882 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:45:58.359745 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346885 4395 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:45:58.359745 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346887 4395 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:45:58.359745 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346890 4395 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:45:58.359745 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346893 4395 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:45:58.359745 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346895 4395 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:45:58.359745 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346898 4395 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:45:58.360338 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346901 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:45:58.360338 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346903 4395 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:45:58.360338 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346906 4395 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:45:58.360338 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346908 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:45:58.360338 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346912 4395 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:45:58.360338 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346915 4395 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:45:58.360338 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346917 4395 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:45:58.360338 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346920 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:45:58.360338 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346922 4395 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:45:58.360338 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346925 4395 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:45:58.360338 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346927 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:45:58.360338 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346930 4395 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:45:58.360338 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346932 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:45:58.360338 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346935 4395 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:45:58.360338 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346937 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:45:58.360338 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346941 4395 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:45:58.360338 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346943 4395 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:45:58.360338 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346946 4395 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:45:58.360338 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346948 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:45:58.360837 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346951 4395 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:45:58.360837 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346953 4395 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:45:58.360837 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346956 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:45:58.360837 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346959 4395 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:45:58.360837 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346963 4395 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:45:58.360837 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346966 4395 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:45:58.360837 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346968 4395 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:45:58.360837 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346971 4395 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:45:58.360837 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346974 4395 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:45:58.360837 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346976 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:45:58.360837 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346979 4395 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:45:58.360837 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346981 4395 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:45:58.360837 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346984 4395 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:45:58.360837 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346986 4395 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:45:58.360837 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346989 4395 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:45:58.360837 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346991 4395 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:45:58.360837 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346994 4395 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:45:58.360837 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.346997 4395 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:45:58.360837 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.347000 4395 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:45:58.360837 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.347002 4395 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:45:58.361349 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.347005 4395 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:45:58.361349 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.347008 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:45:58.361349 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.347010 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:45:58.361349 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.347013 4395 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:45:58.361349 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.347015 4395 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:45:58.361349 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.347018 4395 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:45:58.361349 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.347021 4395 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:45:58.361349 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.347023 4395 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:45:58.361349 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.347026 4395 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:45:58.361349 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.347028 4395 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:45:58.361349 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.347031 4395 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:45:58.361349 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.347034 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:45:58.361349 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.347037 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:45:58.361349 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.347039 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:45:58.361349 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.350586 4395 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 04 04:45:58.361349 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.350600 4395 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 04 04:45:58.361349 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.350619 4395 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 04 04:45:58.361349 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.350624 4395 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 04 04:45:58.361349 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350699 4395 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:45:58.361813 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350705 4395 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:45:58.361813 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350708 4395 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:45:58.361813 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350711 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:45:58.361813 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350714 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:45:58.361813 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350718 4395 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:45:58.361813 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350720 4395 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:45:58.361813 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350723 4395 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:45:58.361813 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350726 4395 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:45:58.361813 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350729 4395 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:45:58.361813 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350732 4395 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:45:58.361813 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350734 4395 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:45:58.361813 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350743 4395 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:45:58.361813 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350746 4395 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:45:58.361813 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350749 4395 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:45:58.361813 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350752 4395 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:45:58.361813 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350754 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:45:58.361813 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350757 4395 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:45:58.361813 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350760 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:45:58.361813 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350763 4395 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:45:58.361813 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350766 4395 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:45:58.362474 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350769 4395 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:45:58.362474 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350771 4395 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:45:58.362474 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350774 4395 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:45:58.362474 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350777 4395 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:45:58.362474 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350779 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:45:58.362474 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350782 4395 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:45:58.362474 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350784 4395 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:45:58.362474 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350787 4395 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:45:58.362474 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350790 4395 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:45:58.362474 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350793 4395 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:45:58.362474 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350795 4395 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:45:58.362474 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350798 4395 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:45:58.362474 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350800 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:45:58.362474 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350803 4395 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:45:58.362474 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350806 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:45:58.362474 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350808 4395 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:45:58.362474 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350811 4395 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:45:58.362474 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350813 4395 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:45:58.362474 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350816 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:45:58.362474 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350818 4395 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:45:58.362962 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350821 4395 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:45:58.362962 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350823 4395 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:45:58.362962 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350826 4395 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:45:58.362962 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350829 4395 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:45:58.362962 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350837 4395 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:45:58.362962 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350840 4395 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:45:58.362962 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350843 4395 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:45:58.362962 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350845 4395 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:45:58.362962 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350848 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:45:58.362962 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350850 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:45:58.362962 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350853 4395 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:45:58.362962 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350856 4395 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:45:58.362962 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350858 4395 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:45:58.362962 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350861 4395 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:45:58.362962 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350863 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:45:58.362962 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350866 4395 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:45:58.362962 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350869 4395 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:45:58.362962 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350872 4395 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:45:58.362962 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350875 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:45:58.363434 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350877 4395 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:45:58.363434 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350880 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:45:58.363434 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350882 4395 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:45:58.363434 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350885 4395 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:45:58.363434 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350887 4395 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:45:58.363434 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350890 4395 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:45:58.363434 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350892 4395 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:45:58.363434 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350895 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:45:58.363434 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350897 4395 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:45:58.363434 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350900 4395 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:45:58.363434 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350902 4395 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:45:58.363434 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350905 4395 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:45:58.363434 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350907 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:45:58.363434 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350910 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:45:58.363434 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350912 4395 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:45:58.363434 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350915 4395 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:45:58.363434 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350918 4395 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:45:58.363434 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350921 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:45:58.363434 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350929 4395 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:45:58.363890 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350934 4395 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:45:58.363890 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350938 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:45:58.363890 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350941 4395 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:45:58.363890 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350944 4395 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:45:58.363890 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350946 4395 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:45:58.363890 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350949 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:45:58.363890 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350952 4395 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:45:58.363890 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350956 4395 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:45:58.363890 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350959 4395 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:45:58.363890 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350962 4395 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:45:58.363890 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350965 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:45:58.363890 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350967 4395 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:45:58.363890 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350970 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:45:58.363890 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350972 4395 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:45:58.363890 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350975 4395 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:45:58.363890 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350978 4395 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:45:58.363890 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350980 4395 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:45:58.363890 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350983 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:45:58.363890 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350986 4395 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:45:58.364380 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350989 4395 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:45:58.364380 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350991 4395 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:45:58.364380 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350994 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:45:58.364380 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350996 4395 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:45:58.364380 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.350999 4395 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:45:58.364380 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351002 4395 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:45:58.364380 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351005 4395 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:45:58.364380 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351007 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:45:58.364380 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.351035 4395 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 04:45:58.364380 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351225 4395 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:45:58.364380 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351229 4395 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:45:58.364380 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351232 4395 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:45:58.364380 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351235 4395 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:45:58.364380 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351243 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:45:58.364380 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351246 4395 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:45:58.364380 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351249 4395 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:45:58.364768 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351251 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:45:58.364768 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351254 4395 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:45:58.364768 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351256 4395 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:45:58.364768 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351264 4395 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:45:58.364768 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351267 4395 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:45:58.364768 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351270 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:45:58.364768 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351272 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:45:58.364768 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351275 4395 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:45:58.364768 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351277 4395 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:45:58.364768 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351280 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:45:58.364768 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351282 4395 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:45:58.364768 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351285 4395 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:45:58.364768 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351287 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:45:58.364768 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351290 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:45:58.364768 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351293 4395 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:45:58.364768 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351310 4395 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:45:58.364768 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351313 4395 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:45:58.364768 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351316 4395 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:45:58.364768 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351318 4395 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:45:58.364768 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351321 4395 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:45:58.365240 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351324 4395 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:45:58.365240 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351326 4395 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:45:58.365240 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351329 4395 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:45:58.365240 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351331 4395 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:45:58.365240 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351334 4395 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:45:58.365240 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351337 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:45:58.365240 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351340 4395 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:45:58.365240 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351342 4395 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:45:58.365240 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351345 4395 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:45:58.365240 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351348 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:45:58.365240 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351357 4395 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:45:58.365240 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351359 4395 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:45:58.365240 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351362 4395 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:45:58.365240 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351364 4395 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:45:58.365240 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351367 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:45:58.365240 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351369 4395 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:45:58.365240 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351372 4395 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:45:58.365240 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351374 4395 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:45:58.365240 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351377 4395 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:45:58.365712 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351379 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:45:58.365712 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351382 4395 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:45:58.365712 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351385 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:45:58.365712 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351387 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:45:58.365712 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351389 4395 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:45:58.365712 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351392 4395 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:45:58.365712 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351394 4395 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:45:58.365712 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351397 4395 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:45:58.365712 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351399 4395 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:45:58.365712 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351402 4395 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:45:58.365712 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351405 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:45:58.365712 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351407 4395 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:45:58.365712 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351409 4395 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:45:58.365712 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351412 4395 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:45:58.365712 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351414 4395 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:45:58.365712 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351417 4395 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:45:58.365712 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351419 4395 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:45:58.365712 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351422 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:45:58.365712 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351425 4395 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:45:58.366154 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351427 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:45:58.366154 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351430 4395 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:45:58.366154 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351432 4395 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:45:58.366154 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351435 4395 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:45:58.366154 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351437 4395 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:45:58.366154 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351447 4395 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:45:58.366154 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351449 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:45:58.366154 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351452 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:45:58.366154 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351455 4395 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:45:58.366154 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351457 4395 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:45:58.366154 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351460 4395 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:45:58.366154 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351462 4395 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:45:58.366154 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351465 4395 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:45:58.366154 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351467 4395 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:45:58.366154 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351470 4395 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:45:58.366154 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351472 4395 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:45:58.366154 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351477 4395 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:45:58.366154 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351480 4395 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:45:58.366154 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351483 4395 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:45:58.366616 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351486 4395 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:45:58.366616 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351488 4395 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:45:58.366616 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351491 4395 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:45:58.366616 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351493 4395 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:45:58.366616 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351496 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:45:58.366616 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351498 4395 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:45:58.366616 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351502 4395 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:45:58.366616 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351505 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:45:58.366616 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351508 4395 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:45:58.366616 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351510 4395 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:45:58.366616 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351513 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:45:58.366616 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351516 4395 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:45:58.366616 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351518 4395 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:45:58.366616 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351520 4395 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:45:58.366616 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351523 4395 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:45:58.366616 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351525 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:45:58.366616 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351528 4395 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:45:58.366616 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351531 4395 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:45:58.366616 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351533 4395 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:45:58.366616 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351541 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:45:58.367085 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351544 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:45:58.367085 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351547 4395 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:45:58.367085 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.351570 4395 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 04:45:58.367085 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351619 4395 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 04:45:58.367085 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351623 4395 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 04:45:58.367085 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351627 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 04:45:58.367085 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351629 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 04:45:58.367085 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351634 4395 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 04:45:58.367085 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351637 4395 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 04:45:58.367085 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351639 4395 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 04:45:58.367085 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351642 4395 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 04:45:58.367085 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351644 4395 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 04:45:58.367085 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351647 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 04:45:58.367085 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351649 4395 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 04:45:58.367085 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351652 4395 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 04:45:58.367085 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351654 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 04:45:58.367472 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351657 4395 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 04:45:58.367472 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351660 4395 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 04:45:58.367472 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351662 4395 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 04:45:58.367472 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351665 4395 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 04:45:58.367472 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351667 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 04:45:58.367472 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351670 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 04:45:58.367472 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351676 4395 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 04:45:58.367472 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351683 4395 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 04:45:58.367472 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351685 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 04:45:58.367472 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351688 4395 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 04:45:58.367472 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351691 4395 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 04:45:58.367472 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351694 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 04:45:58.367472 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351696 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 04:45:58.367472 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351699 4395 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 04:45:58.367472 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351701 4395 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 04:45:58.367472 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351704 4395 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 04:45:58.367472 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351712 4395 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 04:45:58.367472 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351715 4395 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 04:45:58.367472 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351717 4395 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 04:45:58.367472 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351720 4395 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 04:45:58.367976 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351722 4395 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 04:45:58.367976 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351725 4395 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 04:45:58.367976 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351727 4395 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 04:45:58.367976 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351731 4395 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 04:45:58.367976 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351734 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 04:45:58.367976 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351737 4395 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 04:45:58.367976 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351739 4395 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 04:45:58.367976 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351742 4395 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 04:45:58.367976 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351744 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 04:45:58.367976 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351747 4395 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 04:45:58.367976 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351749 4395 feature_gate.go:386] unrecognized feature gate: Example Nov 04 04:45:58.367976 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351752 4395 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 04:45:58.367976 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351755 4395 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 04:45:58.367976 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351757 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 04:45:58.367976 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351760 4395 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 04:45:58.367976 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351762 4395 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 04:45:58.367976 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351764 4395 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 04:45:58.367976 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351767 4395 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 04:45:58.367976 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351770 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 04:45:58.368460 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351773 4395 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 04:45:58.368460 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351776 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 04:45:58.368460 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351779 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 04:45:58.368460 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351782 4395 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 04:45:58.368460 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351784 4395 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 04:45:58.368460 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351787 4395 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 04:45:58.368460 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351789 4395 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 04:45:58.368460 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351792 4395 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 04:45:58.368460 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351794 4395 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 04:45:58.368460 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351797 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 04:45:58.368460 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351806 4395 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 04:45:58.368460 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351809 4395 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 04:45:58.368460 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351811 4395 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 04:45:58.368460 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351814 4395 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 04:45:58.368460 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351817 4395 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 04:45:58.368460 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351819 4395 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 04:45:58.368460 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351823 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 04:45:58.368460 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351825 4395 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 04:45:58.368460 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351828 4395 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 04:45:58.369063 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351830 4395 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 04:45:58.369063 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351833 4395 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 04:45:58.369063 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351835 4395 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 04:45:58.369063 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351838 4395 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 04:45:58.369063 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351841 4395 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 04:45:58.369063 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351843 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 04:45:58.369063 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351846 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 04:45:58.369063 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351848 4395 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 04:45:58.369063 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351851 4395 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 04:45:58.369063 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351854 4395 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 04:45:58.369063 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351857 4395 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 04:45:58.369063 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351859 4395 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 04:45:58.369063 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351861 4395 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 04:45:58.369063 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351866 4395 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 04:45:58.369063 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351868 4395 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 04:45:58.369063 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351873 4395 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 04:45:58.369063 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351876 4395 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 04:45:58.369063 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351879 4395 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 04:45:58.369063 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351881 4395 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 04:45:58.369852 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351884 4395 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 04:45:58.369852 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351887 4395 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 04:45:58.369852 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351889 4395 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 04:45:58.369852 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351892 4395 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 04:45:58.369852 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351894 4395 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 04:45:58.369852 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351903 4395 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 04:45:58.369852 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351907 4395 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 04:45:58.369852 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351911 4395 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 04:45:58.369852 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351913 4395 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 04:45:58.369852 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351916 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 04:45:58.369852 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351919 4395 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 04:45:58.369852 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351922 4395 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 04:45:58.369852 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351924 4395 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 04:45:58.369852 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351927 4395 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 04:45:58.369852 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351929 4395 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 04:45:58.369852 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:45:58.351932 4395 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 04:45:58.369852 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.351983 4395 server.go:957] "Client rotation is on, will bootstrap in background" Nov 04 04:45:58.369852 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.354432 4395 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 04 04:45:58.369852 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.354480 4395 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 04 04:45:58.370746 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.354664 4395 server.go:1014] "Starting client certificate rotation" Nov 04 04:45:58.370746 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.354809 4395 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 04:45:58.370746 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.354863 4395 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-03 04:40:16 +0000 UTC" deadline="2028-07-27 13:08:01.012222856 +0000 UTC" Nov 04 04:45:58.370746 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.354901 4395 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15152h22m2.657325971s" Nov 04 04:45:58.370746 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.355101 4395 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 04:45:58.370746 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.356165 4395 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 04:45:58.370746 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.357678 4395 log.go:25] "Validated CRI v1 runtime API" Nov 04 04:45:58.370746 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.359213 4395 log.go:25] "Validated CRI v1 image API" Nov 04 04:45:58.370746 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.359908 4395 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 04 04:45:58.370746 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.361230 4395 fs.go:135] Filesystem UUIDs: map[0a8d1a56-04db-40ac-965d-9ac5017686fe:/dev/nvme0n1p4 74c82a04-1aa8-43b9-a324-498152680015:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Nov 04 04:45:58.371048 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.361250 4395 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/23b43177dc4829aaf4c28707186c6ac0c41d50126821789ce3dd9cae71a77721/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/23b43177dc4829aaf4c28707186c6ac0c41d50126821789ce3dd9cae71a77721/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/24eecffe0d5a3df21a305e829689cc49da7665f450c7ac5084833a0a390efd7b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/24eecffe0d5a3df21a305e829689cc49da7665f450c7ac5084833a0a390efd7b/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/39dafa117e7beae7f40ed505d989b9fe9b6b2ae29b88f4da370722826eaad31e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/39dafa117e7beae7f40ed505d989b9fe9b6b2ae29b88f4da370722826eaad31e/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/52b6671ae663b31c624bd621cb13d966ee18207be2a20611890ab2f9c1b70b08/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/52b6671ae663b31c624bd621cb13d966ee18207be2a20611890ab2f9c1b70b08/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/59837c93d67015f723085d6ae47d5e6e7c8a810f6bd904ef7641e6d6eee870d2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/59837c93d67015f723085d6ae47d5e6e7c8a810f6bd904ef7641e6d6eee870d2/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5a32a8e915e44453cceb3a6c6b4f03f2445cffc4f9c880e999a1c353ab64db22/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5a32a8e915e44453cceb3a6c6b4f03f2445cffc4f9c880e999a1c353ab64db22/userdata/shm major:0 minor:201 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5ca6e5771579d93b64701a87510bbf7d2d64d3e6cf58bb36c4633844df0a5081/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5ca6e5771579d93b64701a87510bbf7d2d64d3e6cf58bb36c4633844df0a5081/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7ccb2c5cf6cf822ca5d1c4c7b51f5f08960eb61b3971ebad33d057addc0bf835/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7ccb2c5cf6cf822ca5d1c4c7b51f5f08960eb61b3971ebad33d057addc0bf835/userdata/shm major:0 minor:204 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9c5ddfa0f2d283e32032e690290a63a1f13dafec60cc2bdac506c6980248ff45/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9c5ddfa0f2d283e32032e690290a63a1f13dafec60cc2bdac506c6980248ff45/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c0105ca9ce2f010d642378a9904480ecf73c2953090017a5d23f612c5aea388f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c0105ca9ce2f010d642378a9904480ecf73c2953090017a5d23f612c5aea388f/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/da6f323aee2ca34cdfc7d0e4609ca9d1830c997a3d52cdb2b1cacbdb73886064/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/da6f323aee2ca34cdfc7d0e4609ca9d1830c997a3d52cdb2b1cacbdb73886064/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dc7d6e11710a5c4826009a9c64aaa572ca406dedd7ab40a50497e27f7bfa3004/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dc7d6e11710a5c4826009a9c64aaa572ca406dedd7ab40a50497e27f7bfa3004/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e9f1723cdddd9a1bac5d41f731e8a1520baad29a0f2ccd976e47f907bf29486b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e9f1723cdddd9a1bac5d41f731e8a1520baad29a0f2ccd976e47f907bf29486b/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31e07d9c-93bc-4e8d-909b-e65406ff62df/volumes/kubernetes.io~projected/kube-api-access-lppk7:{mountpoint:/var/lib/kubelet/pods/31e07d9c-93bc-4e8d-909b-e65406ff62df/volumes/kubernetes.io~projected/kube-api-access-lppk7 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e3bf36f-7156-4151-9d6d-0937169dbcca/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/3e3bf36f-7156-4151-9d6d-0937169dbcca/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/52f02c25-48ed-46dc-82bc-23016ffa595d/volumes/kubernetes.io~projected/kube-api-access-b7vxq:{mountpoint:/var/lib/kubelet/pods/52f02c25-48ed-46dc-82bc-23016ffa595d/volumes/kubernetes.io~projected/kube-api-access-b7vxq major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59d18119-72a5-410f-be97-90613af651bd/volumes/kubernetes.io~projected/kube-api-access-b7jdp:{mountpoint:/var/lib/kubelet/pods/59d18119-72a5-410f-be97-90613af651bd/volumes/kubernetes.io~projected/kube-api-access-b7jdp major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c1153d6-83db-4eb7-95e0-78d09bd61d6d/volumes/kubernetes.io~projected/kube-api-access-5d9jw:{mountpoint:/var/lib/kubelet/pods/5c1153d6-83db-4eb7-95e0-78d09bd61d6d/volumes/kubernetes.io~projected/kube-api-access-5d9jw major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7380461b-403d-4470-8c1c-6433616b1dd2/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/7380461b-403d-4470-8c1c-6433616b1dd2/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7380461b-403d-4470-8c1c-6433616b1dd2/volumes/kubernetes.io~projected/kube-api-access-pwmfd:{mountpoint:/var/lib/kubelet/pods/7380461b-403d-4470-8c1c-6433616b1dd2/volumes/kubernetes.io~projected/kube-api-access-pwmfd major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7380461b-403d-4470-8c1c-6433616b1dd2/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/7380461b-403d-4470-8c1c-6433616b1dd2/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a/volumes/kubernetes.io~projected/kube-api-access-gchdh:{mountpoint:/var/lib/kubelet/pods/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a/volumes/kubernetes.io~projected/kube-api-access-gchdh major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643/volumes/kubernetes.io~empty-dir/tmp major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643/volumes/kubernetes.io~projected/kube-api-access-kp26f:{mountpoint:/var/lib/kubelet/pods/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643/volumes/kubernetes.io~projected/kube-api-access-kp26f major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7936867-0a26-4c2e-b94d-a1756cd14873/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/a7936867-0a26-4c2e-b94d-a1756cd14873/volumes/kubernetes.io~secret/agent-certs major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b/volumes/kubernetes.io~projected/kube-api-access-w6kb6:{mountpoint:/var/lib/kubelet/pods/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b/volumes/kubernetes.io~projected/kube-api-access-w6kb6 major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cab63986-8b41-42c4-b7b0-4438e54fa424/volumes/kubernetes.io~projected/kube-api-access-zszmj:{mountpoint:/var/lib/kubelet/pods/cab63986-8b41-42c4-b7b0-4438e54fa424/volumes/kubernetes.io~projected/kube-api-access-zszmj major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0c260a1-2a73-4b2b-b6a7-2f9717941a1b/volumes/kubernetes.io~projected/kube-api-access-pd5qw:{mountpoint:/var/lib/kubelet/pods/f0c260a1-2a73-4b2b-b6a7-2f9717941a1b/volumes/kubernetes.io~projected/kube-api-access-pd5qw major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0c260a1-2a73-4b2b-b6a7-2f9717941a1b/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/f0c260a1-2a73-4b2b-b6a7-2f9717941a1b/volumes/kubernetes.io~secret/metrics-certs major:0 minor:45 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/92acb051113d4e35a0c363d4d58c1440021a2d7e3b71f6b4dd81d29863d0e43f/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/76c30aa81c67789271ab53acdd69031241677f5ec143f7d881f3367827d1b1e7/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/427b69fee7e73f1cdcda66b27bedbce23089d75024631de051798e42242b6923/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/0db69d24f4776ce765ca41775dd8f83bd8e6d0650009106031c2d690236a6115/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/77e15161ea3c19c8fefdf7ee091d58fe92f79cae8c4cabb19d6d3aae49d65ae8/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/8bcea09cff08467549ccf4a6b86bad70dda1d24bcfefca6bcaa124906c834a9c/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/ce464d6a6f33fb47db656ead1e479aab2978b81a9e1e133d012fec19a418f819/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/89ac84eea238b9a985568e24db4e6420160902b797dabdde4c8c2b9a0ac575d2/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/0938834d9fe4253616113a27ead358d6a37feb5cb37d9f58f407191c0b34ce53/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/95bd68a97f5259b41f0eebe1b7ae268d45ccb8689c360d12395faa7736ddb3f1/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/adfd97628fb44aacaa00f750a465353cb018fc161f83df86b31303be4b6f1ad9/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/768df29f1a2da1ef6a43dd8c6ca5ec88b43a98277c34f3882f91de8845ac047f/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/fe4ee19e429fcdaa5f341d87d2321f3faede05c162ec38adfbba59e81db58ab4/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/7800f44be9486a8a4f6471ca6646644c9c3758388b93023ce3eb31bef3a0ebc0/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/4a84e6011142885659aea872a75aab478c8eba1c0c8a8b33d010271d081273d9/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/273a7c825b0673d2924d3c2bea641949184f3a9298317eff903be0e33216152d/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/d32cbc721cf6a9b4d2eddbaef2a7452d12a5b97948c3b989b57ed30418a5191c/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/771e3996931b386870ef6a49732fc75645cc088e28398c08acdc14ccda4877b9/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/c52206732423496c73ae1b7aca6ae8034f4157125e68ae9ec4864efc1942d2b6/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/d8d827540f0085a65b8be44d43b80696c00e31e6f32240592a52e22fa1e45f1b/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/a7a8ac952f38c07cc9b8a6b52c5ec61f41c0beb48d6f069c733c634fdb35c2bc/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-208:{mountpoint:/var/lib/containers/storage/overlay/c0a5b9d2c18dde6a250667c8355a6957835ddfbd3baef681b8dd4f95297845a8/merged major:0 minor:208 fsType:overlay blockSize:0} overlay_0-210:{mountpoint:/var/lib/containers/storage/overlay/926565b067443c983d2c5f373f0dcfea0ff035d527e2e7c34819396c42d4f68a/merged major:0 minor:210 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/dc3591f56a5fbb0d7813db424248f36b6252caa79d55c24c31d5cd6f49082efe/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/ddec8ea21dcf56dedaaba71530d05114ae8c0aa07ec13aa3c548bc46402abee3/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/a55e9eedf6856ab8103a85d1e37e5ba8e7cb146ba279cf886dfa62bb9ea01e55/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/540d388b1be28d208922d27186460dfa2f4d49fb17630c889568452be6bc6c56/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-227:{mountpoint:/var/lib/containers/storage/overlay/fa95736470e938bfe7f3e5c26c54ccef3893c3e042c30138b40b9f2164edc64d/merged major:0 minor:227 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/0ab15856eb9bcc4e2e264b1c5a03be8cbb5342a81d3de5e130ba6316600e8d01/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/c7bd539808262a743b52fc38810bcbb610035a993b0fdceb123d2135563627cd/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/8baa941b9caafecac264a54457c74984a6b237900a0e775e9516af758c467d83/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/0be11d6f9ae935eaece7cb8492c7e0261df064077f64d2409c33b68de008a105/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/408a55b52cffa10ce583275212f0db57d6cca8eae0a3545f53b398790ac24d2a/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/946f0f69496c2542f312813910e476dea4056c7df27f0d32bf3b0735d2ddb30d/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/a2cf3fe99ae9385d725ae2ab8b5a1aabf342bb3c5ced4930d7a7689ed5b169b1/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/712cc4b81bcc097d34991e30e6160c9bb3bd3f349475698814b84fc08dbe5a7a/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/54c08b8bc3398d3201cc8b399585c941e9090590d76d76d5229a71b57761bd05/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/57a6081e0dcc9622340129e25b1866f827802994f7e3cb9ae50d4fbe58f36f83/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 04 04:45:58.371048 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.367479 4395 manager.go:217] Machine: {Timestamp:2026-11-04 04:45:58.36702086 +0000 UTC m=+0.057103300 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098123 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c026f93b8c85f8b97ef083c4a8115 SystemUUID:ec2c026f-93b8-c85f-8b97-ef083c4a8115 BootID:e21ebccf-c7d6-4baf-9c55-6579bf3f2121 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/24eecffe0d5a3df21a305e829689cc49da7665f450c7ac5084833a0a390efd7b/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-208 DeviceMajor:0 DeviceMinor:208 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e9f1723cdddd9a1bac5d41f731e8a1520baad29a0f2ccd976e47f907bf29486b/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/da6f323aee2ca34cdfc7d0e4609ca9d1830c997a3d52cdb2b1cacbdb73886064/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/52f02c25-48ed-46dc-82bc-23016ffa595d/volumes/kubernetes.io~projected/kube-api-access-b7vxq DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5ca6e5771579d93b64701a87510bbf7d2d64d3e6cf58bb36c4633844df0a5081/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7ccb2c5cf6cf822ca5d1c4c7b51f5f08960eb61b3971ebad33d057addc0bf835/userdata/shm DeviceMajor:0 DeviceMinor:204 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/7380461b-403d-4470-8c1c-6433616b1dd2/volumes/kubernetes.io~projected/kube-api-access-pwmfd DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c0105ca9ce2f010d642378a9904480ecf73c2953090017a5d23f612c5aea388f/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-227 DeviceMajor:0 DeviceMinor:227 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/59837c93d67015f723085d6ae47d5e6e7c8a810f6bd904ef7641e6d6eee870d2/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/39dafa117e7beae7f40ed505d989b9fe9b6b2ae29b88f4da370722826eaad31e/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b/volumes/kubernetes.io~projected/kube-api-access-w6kb6 DeviceMajor:0 DeviceMinor:50 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9c5ddfa0f2d283e32032e690290a63a1f13dafec60cc2bdac506c6980248ff45/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7380461b-403d-4470-8c1c-6433616b1dd2/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/59d18119-72a5-410f-be97-90613af651bd/volumes/kubernetes.io~projected/kube-api-access-b7jdp DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643/volumes/kubernetes.io~projected/kube-api-access-kp26f DeviceMajor:0 DeviceMinor:53 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/31e07d9c-93bc-4e8d-909b-e65406ff62df/volumes/kubernetes.io~projected/kube-api-access-lppk7 DeviceMajor:0 DeviceMinor:51 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a/volumes/kubernetes.io~projected/kube-api-access-gchdh DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dc7d6e11710a5c4826009a9c64aaa572ca406dedd7ab40a50497e27f7bfa3004/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/5c1153d6-83db-4eb7-95e0-78d09bd61d6d/volumes/kubernetes.io~projected/kube-api-access-5d9jw DeviceMajor:0 DeviceMinor:52 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a7936867-0a26-4c2e-b94d-a1756cd14873/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:54 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f0c260a1-2a73-4b2b-b6a7-2f9717941a1b/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7380461b-403d-4470-8c1c-6433616b1dd2/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/23b43177dc4829aaf4c28707186c6ac0c41d50126821789ce3dd9cae71a77721/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/52b6671ae663b31c624bd621cb13d966ee18207be2a20611890ab2f9c1b70b08/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:47 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cab63986-8b41-42c4-b7b0-4438e54fa424/volumes/kubernetes.io~projected/kube-api-access-zszmj DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5a32a8e915e44453cceb3a6c6b4f03f2445cffc4f9c880e999a1c353ab64db22/userdata/shm DeviceMajor:0 DeviceMinor:201 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f0c260a1-2a73-4b2b-b6a7-2f9717941a1b/volumes/kubernetes.io~projected/kube-api-access-pd5qw DeviceMajor:0 DeviceMinor:49 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3e3bf36f-7156-4151-9d6d-0937169dbcca/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-210 DeviceMajor:0 DeviceMinor:210 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:39dafa117e7beae MacAddress:72:59:8a:c9:1f:20 Speed:10000 Mtu:8901} {Name:5a32a8e915e4445 MacAddress:da:51:aa:6e:bc:e4 Speed:10000 Mtu:8901} {Name:7ccb2c5cf6cf822 MacAddress:ee:7f:e5:02:8c:32 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:f3:20:b0:db Speed:0 Mtu:9001} {Name:br-int MacAddress:26:58:3d:2d:19:dc Speed:0 Mtu:8901} {Name:ens5 MacAddress:06:ff:f3:20:b0:db Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:06:2e:41:0b:c3:d3 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:b6:36:f1:f6:c7:b9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 04 04:45:58.371048 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.367704 4395 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 04 04:45:58.371048 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.367740 4395 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 04 04:45:58.371048 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.368007 4395 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 04 04:45:58.371399 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.368029 4395 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-224.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 04 04:45:58.371399 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.368150 4395 topology_manager.go:143] "Creating topology manager with none policy" Nov 04 04:45:58.371399 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.368158 4395 container_manager_linux.go:311] "Creating device plugin manager" Nov 04 04:45:58.371399 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.368172 4395 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 04:45:58.371399 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.368190 4395 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 04:45:58.371399 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.368262 4395 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 04 04:45:58.371399 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.368410 4395 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 04 04:45:58.371399 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.368519 4395 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 04 04:45:58.371399 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.368556 4395 kubelet.go:493] "Attempting to sync node with API server" Nov 04 04:45:58.371399 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.368568 4395 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 04 04:45:58.371399 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.368577 4395 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 04 04:45:58.371399 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.368585 4395 kubelet.go:405] "Adding apiserver pod source" Nov 04 04:45:58.371399 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.368592 4395 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 04 04:45:58.371399 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.369598 4395 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 04 04:45:58.371399 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.369613 4395 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 04:45:58.371399 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.370150 4395 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 04 04:45:58.373169 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.373153 4395 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 04 04:45:58.373211 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.373171 4395 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 04:45:58.373314 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.373288 4395 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 04 04:45:58.373546 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.373531 4395 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 04 04:45:58.373589 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.373579 4395 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 04 04:45:58.373696 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.373685 4395 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 04 04:45:58.373747 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.373708 4395 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 04 04:45:58.373747 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.373722 4395 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 04 04:45:58.373747 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.373733 4395 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 04 04:45:58.373747 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.373744 4395 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 04 04:45:58.373879 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.373752 4395 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 04 04:45:58.373879 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.373763 4395 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 04 04:45:58.373879 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.373771 4395 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 04 04:45:58.373879 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.373781 4395 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 04 04:45:58.373879 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.373790 4395 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 04 04:45:58.373879 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.373803 4395 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 04 04:45:58.373879 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.373815 4395 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 04 04:45:58.373879 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.373852 4395 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 04 04:45:58.373879 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.373860 4395 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 04 04:45:58.376943 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.376930 4395 server.go:1263] "Started kubelet" Nov 04 04:45:58.377029 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.377003 4395 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 04 04:45:58.377091 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.377026 4395 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 04 04:45:58.377143 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.377096 4395 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 04 04:45:58.377423 ip-10-0-134-224 systemd[1]: Started Kubernetes Kubelet. Nov 04 04:45:58.377593 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.377515 4395 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 04 04:45:58.378374 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.378358 4395 server.go:317] "Adding debug handlers to kubelet server" Nov 04 04:45:58.379837 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.379819 4395 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 04 04:45:58.380705 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.379843 4395 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 04 04:45:58.380705 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:45:58.379934 4395 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-224.us-west-2.compute.internal\" not found" Nov 04 04:45:58.380705 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.379920 4395 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 04:40:17 +0000 UTC" deadline="2028-03-30 16:15:54.507218829 +0000 UTC" Nov 04 04:45:58.380705 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.379969 4395 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12299h29m56.127253811s" Nov 04 04:45:58.380705 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.380703 4395 factory.go:55] Registering systemd factory Nov 04 04:45:58.380991 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.380718 4395 factory.go:223] Registration of the systemd container factory successfully Nov 04 04:45:58.380991 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.380909 4395 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 04 04:45:58.380991 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.380961 4395 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 04 04:45:58.380991 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.380974 4395 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 04 04:45:58.382397 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.382275 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7936867-0a26-4c2e-b94d-a1756cd14873" volumeName="kubernetes.io/secret/a7936867-0a26-4c2e-b94d-a1756cd14873-agent-certs" seLinuxMountContext="" Nov 04 04:45:58.382397 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.382335 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc352ca6-4a2d-4f7f-a434-7c45dd704e4b" volumeName="kubernetes.io/configmap/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-cni-binary-copy" seLinuxMountContext="" Nov 04 04:45:58.382397 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.382351 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0c260a1-2a73-4b2b-b6a7-2f9717941a1b" volumeName="kubernetes.io/projected/f0c260a1-2a73-4b2b-b6a7-2f9717941a1b-kube-api-access-pd5qw" seLinuxMountContext="" Nov 04 04:45:58.382397 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.382366 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31e07d9c-93bc-4e8d-909b-e65406ff62df" volumeName="kubernetes.io/configmap/31e07d9c-93bc-4e8d-909b-e65406ff62df-cni-binary-copy" seLinuxMountContext="" Nov 04 04:45:58.382397 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.382379 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52f02c25-48ed-46dc-82bc-23016ffa595d" volumeName="kubernetes.io/empty-dir/52f02c25-48ed-46dc-82bc-23016ffa595d-tmp-dir" seLinuxMountContext="" Nov 04 04:45:58.382397 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.382393 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c1153d6-83db-4eb7-95e0-78d09bd61d6d" volumeName="kubernetes.io/configmap/5c1153d6-83db-4eb7-95e0-78d09bd61d6d-serviceca" seLinuxMountContext="" Nov 04 04:45:58.382743 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.382407 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c1153d6-83db-4eb7-95e0-78d09bd61d6d" volumeName="kubernetes.io/projected/5c1153d6-83db-4eb7-95e0-78d09bd61d6d-kube-api-access-5d9jw" seLinuxMountContext="" Nov 04 04:45:58.382743 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.382421 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7380461b-403d-4470-8c1c-6433616b1dd2" volumeName="kubernetes.io/configmap/7380461b-403d-4470-8c1c-6433616b1dd2-ovnkube-config" seLinuxMountContext="" Nov 04 04:45:58.382743 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.382435 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7380461b-403d-4470-8c1c-6433616b1dd2" volumeName="kubernetes.io/projected/7380461b-403d-4470-8c1c-6433616b1dd2-kube-api-access-pwmfd" seLinuxMountContext="" Nov 04 04:45:58.382743 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.382447 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7380461b-403d-4470-8c1c-6433616b1dd2" volumeName="kubernetes.io/secret/7380461b-403d-4470-8c1c-6433616b1dd2-ovn-node-metrics-cert" seLinuxMountContext="" Nov 04 04:45:58.382743 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.382460 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ae0b6cf-e85a-40a1-b61b-01bd5fde3643" volumeName="kubernetes.io/empty-dir/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-tmp" seLinuxMountContext="" Nov 04 04:45:58.382743 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.382473 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52f02c25-48ed-46dc-82bc-23016ffa595d" volumeName="kubernetes.io/projected/52f02c25-48ed-46dc-82bc-23016ffa595d-kube-api-access-b7vxq" seLinuxMountContext="" Nov 04 04:45:58.382743 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.382485 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7422e4c1-59b2-4bdc-8fb1-7542ba692a6a" volumeName="kubernetes.io/projected/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-kube-api-access-gchdh" seLinuxMountContext="" Nov 04 04:45:58.382743 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.382498 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ae0b6cf-e85a-40a1-b61b-01bd5fde3643" volumeName="kubernetes.io/empty-dir/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-tuned" seLinuxMountContext="" Nov 04 04:45:58.382743 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.382513 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ae0b6cf-e85a-40a1-b61b-01bd5fde3643" volumeName="kubernetes.io/projected/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-kube-api-access-kp26f" seLinuxMountContext="" Nov 04 04:45:58.382743 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.382525 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7936867-0a26-4c2e-b94d-a1756cd14873" volumeName="kubernetes.io/configmap/a7936867-0a26-4c2e-b94d-a1756cd14873-konnectivity-ca" seLinuxMountContext="" Nov 04 04:45:58.382743 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.382546 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc352ca6-4a2d-4f7f-a434-7c45dd704e4b" volumeName="kubernetes.io/configmap/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-multus-daemon-config" seLinuxMountContext="" Nov 04 04:45:58.382743 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.382562 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cab63986-8b41-42c4-b7b0-4438e54fa424" volumeName="kubernetes.io/projected/cab63986-8b41-42c4-b7b0-4438e54fa424-kube-api-access-zszmj" seLinuxMountContext="" Nov 04 04:45:58.382743 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.382575 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0c260a1-2a73-4b2b-b6a7-2f9717941a1b" volumeName="kubernetes.io/secret/f0c260a1-2a73-4b2b-b6a7-2f9717941a1b-metrics-certs" seLinuxMountContext="" Nov 04 04:45:58.382743 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.382586 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31e07d9c-93bc-4e8d-909b-e65406ff62df" volumeName="kubernetes.io/configmap/31e07d9c-93bc-4e8d-909b-e65406ff62df-cni-sysctl-allowlist" seLinuxMountContext="" Nov 04 04:45:58.382743 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.382597 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31e07d9c-93bc-4e8d-909b-e65406ff62df" volumeName="kubernetes.io/configmap/31e07d9c-93bc-4e8d-909b-e65406ff62df-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 04 04:45:58.382743 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.382609 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31e07d9c-93bc-4e8d-909b-e65406ff62df" volumeName="kubernetes.io/projected/31e07d9c-93bc-4e8d-909b-e65406ff62df-kube-api-access-lppk7" seLinuxMountContext="" Nov 04 04:45:58.382743 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.382632 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e3bf36f-7156-4151-9d6d-0937169dbcca" volumeName="kubernetes.io/secret/3e3bf36f-7156-4151-9d6d-0937169dbcca-original-pull-secret" seLinuxMountContext="" Nov 04 04:45:58.382743 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.382644 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59d18119-72a5-410f-be97-90613af651bd" volumeName="kubernetes.io/projected/59d18119-72a5-410f-be97-90613af651bd-kube-api-access-b7jdp" seLinuxMountContext="" Nov 04 04:45:58.382743 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.382657 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc352ca6-4a2d-4f7f-a434-7c45dd704e4b" volumeName="kubernetes.io/projected/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-kube-api-access-w6kb6" seLinuxMountContext="" Nov 04 04:45:58.382743 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.382668 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cab63986-8b41-42c4-b7b0-4438e54fa424" volumeName="kubernetes.io/configmap/cab63986-8b41-42c4-b7b0-4438e54fa424-iptables-alerter-script" seLinuxMountContext="" Nov 04 04:45:58.382743 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.382681 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7380461b-403d-4470-8c1c-6433616b1dd2" volumeName="kubernetes.io/configmap/7380461b-403d-4470-8c1c-6433616b1dd2-env-overrides" seLinuxMountContext="" Nov 04 04:45:58.382743 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.382692 4395 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7380461b-403d-4470-8c1c-6433616b1dd2" volumeName="kubernetes.io/configmap/7380461b-403d-4470-8c1c-6433616b1dd2-ovnkube-script-lib" seLinuxMountContext="" Nov 04 04:45:58.382743 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.382703 4395 reconstruct.go:97] "Volume reconstruction finished" Nov 04 04:45:58.382743 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.382711 4395 reconciler.go:29] "Reconciler: start to sync state" Nov 04 04:45:58.384207 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.384190 4395 factory.go:153] Registering CRI-O factory Nov 04 04:45:58.384291 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.384210 4395 factory.go:223] Registration of the crio container factory successfully Nov 04 04:45:58.385256 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.385240 4395 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 04 04:45:58.385727 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.385267 4395 factory.go:103] Registering Raw factory Nov 04 04:45:58.385727 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.385285 4395 manager.go:1199] Started watching for new ooms in manager Nov 04 04:45:58.385832 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.385810 4395 manager.go:319] Starting recovery of all containers Nov 04 04:45:58.407046 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.406988 4395 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 04 04:45:58.408759 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.408729 4395 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 04 04:45:58.408759 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.408760 4395 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 04 04:45:58.408918 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.408785 4395 kubelet.go:2521] "Starting kubelet main sync loop" Nov 04 04:45:58.408918 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:45:58.408829 4395 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 04 04:45:58.415035 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.415009 4395 generic.go:356] "Generic (PLEG): container finished" podID="95a50811676d3438ddaa64148db61400" containerID="50f109a8342c11809dea05e856eb8ec071a03f0979c626a99e4dedb7fcd3e0da" exitCode=0 Nov 04 04:45:58.438631 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.438606 4395 generic.go:356] "Generic (PLEG): container finished" podID="31e07d9c-93bc-4e8d-909b-e65406ff62df" containerID="bae6b8fc7b6ca1a06d2c068068f04c091ba31336a9405402b2e9582a5d5f40fc" exitCode=0 Nov 04 04:45:58.438631 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.438630 4395 generic.go:356] "Generic (PLEG): container finished" podID="31e07d9c-93bc-4e8d-909b-e65406ff62df" containerID="bdc7189a4b9c6ef788f0624785d437016f0f231808bec2e065b2727d2a76637c" exitCode=0 Nov 04 04:45:58.438781 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.438639 4395 generic.go:356] "Generic (PLEG): container finished" podID="31e07d9c-93bc-4e8d-909b-e65406ff62df" containerID="486a127530789d1f203e7868d4b26167e4e57bef4f14da888832d7cbe477efa3" exitCode=0 Nov 04 04:45:58.438781 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.438649 4395 generic.go:356] "Generic (PLEG): container finished" podID="31e07d9c-93bc-4e8d-909b-e65406ff62df" containerID="f1ed44958c2fa0725dcb552eb33bc9cdf5083cf216adc03e079e32fdd8a1afcd" exitCode=0 Nov 04 04:45:58.438781 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.438658 4395 generic.go:356] "Generic (PLEG): container finished" podID="31e07d9c-93bc-4e8d-909b-e65406ff62df" containerID="5369c00880cb269e292e9a796b5eed7b98e49708007bb73a970daf58ffde2caf" exitCode=0 Nov 04 04:45:58.438781 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.438663 4395 generic.go:356] "Generic (PLEG): container finished" podID="31e07d9c-93bc-4e8d-909b-e65406ff62df" containerID="d72d5c0598318327eaad79bb5dcd047d76f14c5f9f8287eb87c43e68ad3ec5f1" exitCode=0 Nov 04 04:45:58.446716 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.446699 4395 manager.go:324] Recovery completed Nov 04 04:45:58.458717 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.458702 4395 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:45:58.459675 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.459660 4395 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:45:58.459738 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.459687 4395 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:45:58.459738 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.459697 4395 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 04:45:58.460811 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.460793 4395 cpu_manager.go:226] "Starting" policy="none" Nov 04 04:45:58.460811 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.460805 4395 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 04 04:45:58.460811 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.460809 4395 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:45:58.461007 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.460820 4395 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 04 04:45:58.461007 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.460917 4395 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 04 04:45:58.461007 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.460927 4395 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 04 04:45:58.461007 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.460942 4395 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 04 04:45:58.461007 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.460947 4395 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 04 04:45:58.461007 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.460952 4395 policy_none.go:50] "Start" Nov 04 04:45:58.461007 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.460960 4395 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 04 04:45:58.461007 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.460968 4395 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 04 04:45:58.461226 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.461090 4395 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 04 04:45:58.461226 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.461097 4395 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 04 04:45:58.461226 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.461103 4395 policy_none.go:44] "Start" Nov 04 04:45:58.461525 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.461509 4395 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:45:58.461587 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.461537 4395 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:45:58.461587 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.461547 4395 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 04:45:58.464780 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.464757 4395 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 04 04:45:58.467155 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.467141 4395 manager.go:341] "Starting Device Plugin manager" Nov 04 04:45:58.467246 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:45:58.467164 4395 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 04 04:45:58.467246 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.467172 4395 server.go:87] "Starting device plugin registration server" Nov 04 04:45:58.467484 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.467470 4395 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 04 04:45:58.467528 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.467485 4395 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 04 04:45:58.467576 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.467561 4395 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 04 04:45:58.467683 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.467650 4395 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 04 04:45:58.467749 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.467691 4395 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 04 04:45:58.467749 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.467698 4395 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 04 04:45:58.467842 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.467729 4395 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-04T04:45:58.467681744Z","UUID":"e1b7a5c4-b7ae-4514-9fa3-870f3905bd96","Handler":null,"Name":"","Endpoint":""} Nov 04 04:45:58.468837 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:45:58.468818 4395 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 04 04:45:58.468916 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:45:58.468854 4395 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-224.us-west-2.compute.internal\" not found" Nov 04 04:45:58.469052 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.469037 4395 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 04 04:45:58.469124 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.469060 4395 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 04 04:45:58.510175 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.509765 4395 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-224.us-west-2.compute.internal"] Nov 04 04:45:58.510274 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.510261 4395 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:45:58.510974 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.510959 4395 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:45:58.511057 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.510986 4395 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:45:58.511057 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.511001 4395 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 04:45:58.511242 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.511229 4395 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:45:58.511381 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.511357 4395 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:45:58.511947 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.511930 4395 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:45:58.511947 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.511942 4395 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:45:58.512059 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.511951 4395 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:45:58.512059 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.511961 4395 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 04:45:58.512059 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.511964 4395 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:45:58.512059 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.511974 4395 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 04:45:58.512172 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:45:58.512080 4395 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-224.us-west-2.compute.internal\" not found" node="ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:58.512172 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.512145 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal" event={"ID":"95a50811676d3438ddaa64148db61400","Type":"ContainerStarted","Data":"aeb3f05d45aebc2e4a633045094aca5234ea1ce5006567bae982477fd1f56dbe"} Nov 04 04:45:58.512230 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.512174 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal" event={"ID":"95a50811676d3438ddaa64148db61400","Type":"ContainerDied","Data":"50f109a8342c11809dea05e856eb8ec071a03f0979c626a99e4dedb7fcd3e0da"} Nov 04 04:45:58.512230 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.512189 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal" event={"ID":"95a50811676d3438ddaa64148db61400","Type":"ContainerStarted","Data":"59837c93d67015f723085d6ae47d5e6e7c8a810f6bd904ef7641e6d6eee870d2"} Nov 04 04:45:58.512290 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.512247 4395 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:45:58.512290 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.512257 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-224.us-west-2.compute.internal" event={"ID":"88d814719e03f487d48b25be7946048f","Type":"ContainerStarted","Data":"e198945739bceb0aa79e60e26855d5dbabb1932df5eaeddb37c371626682dd03"} Nov 04 04:45:58.512290 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.512276 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-224.us-west-2.compute.internal" event={"ID":"88d814719e03f487d48b25be7946048f","Type":"ContainerStarted","Data":"e9f1723cdddd9a1bac5d41f731e8a1520baad29a0f2ccd976e47f907bf29486b"} Nov 04 04:45:58.512898 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.512883 4395 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:45:58.512981 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.512905 4395 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:45:58.512981 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.512913 4395 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 04:45:58.513057 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:45:58.513011 4395 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-224.us-west-2.compute.internal\" not found" node="ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:58.568229 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.568204 4395 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 04:45:58.568961 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.568945 4395 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 04:45:58.569018 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.568972 4395 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 04:45:58.569018 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.568983 4395 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 04:45:58.569018 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.568997 4395 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:58.584817 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.584799 4395 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 04 04:45:58.614458 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.614437 4395 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 04:45:58.624384 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.624338 4395 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 04:45:58.624556 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.624541 4395 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 04:45:58.672418 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.672392 4395 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 04:45:58.680241 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.680219 4395 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:58.685227 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.685210 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/95a50811676d3438ddaa64148db61400-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal\" (UID: \"95a50811676d3438ddaa64148db61400\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:58.685293 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.685233 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95a50811676d3438ddaa64148db61400-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal\" (UID: \"95a50811676d3438ddaa64148db61400\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:58.685293 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.685250 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/88d814719e03f487d48b25be7946048f-config\") pod \"kube-apiserver-proxy-ip-10-0-134-224.us-west-2.compute.internal\" (UID: \"88d814719e03f487d48b25be7946048f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:58.720379 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.720348 4395 nodeinfomanager.go:471] existing CSINode "ip-10-0-134-224.us-west-2.compute.internal" is owned by different node (oldNodeID="df9584bd-e49e-425e-bc53-5be234f42c6d", newNodeID=""), cleaning up... Nov 04 04:45:58.720622 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.720605 4395 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:58.720662 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.720653 4395 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:58.721031 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.721017 4395 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeNotReady" Nov 04 04:45:58.721075 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.721043 4395 setters.go:546] "Node became not ready" node="ip-10-0-134-224.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-04T04:45:58Z","lastTransitionTime":"2026-11-04T04:45:58Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 04 04:45:58.736430 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.736411 4395 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 04:45:58.736503 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:45:58.736445 4395 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:58.736503 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.736455 4395 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:58.785762 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.785742 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/88d814719e03f487d48b25be7946048f-config\") pod \"kube-apiserver-proxy-ip-10-0-134-224.us-west-2.compute.internal\" (UID: \"88d814719e03f487d48b25be7946048f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:58.785833 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.785768 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/95a50811676d3438ddaa64148db61400-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal\" (UID: \"95a50811676d3438ddaa64148db61400\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:58.785833 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.785785 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95a50811676d3438ddaa64148db61400-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal\" (UID: \"95a50811676d3438ddaa64148db61400\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:58.785902 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.785831 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95a50811676d3438ddaa64148db61400-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal\" (UID: \"95a50811676d3438ddaa64148db61400\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:58.785902 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.785843 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/88d814719e03f487d48b25be7946048f-config\") pod \"kube-apiserver-proxy-ip-10-0-134-224.us-west-2.compute.internal\" (UID: \"88d814719e03f487d48b25be7946048f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:58.785902 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.785853 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/95a50811676d3438ddaa64148db61400-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal\" (UID: \"95a50811676d3438ddaa64148db61400\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:58.791127 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:58.791110 4395 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 04:45:58.791190 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:45:58.791142 4395 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-224.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:59.369764 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.369741 4395 apiserver.go:52] "Watching apiserver" Nov 04 04:45:59.443889 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.443865 4395 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:59.444017 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.444001 4395 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:59.465859 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.465836 4395 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 04 04:45:59.466226 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.466203 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-ct2mt","openshift-dns/node-resolver-7f22t","openshift-image-registry/node-ca-7tlqm","openshift-multus/multus-additional-cni-plugins-p9jxj","kube-system/global-pull-secret-syncer-mwgm5","openshift-cluster-node-tuning-operator/tuned-dj7mc","openshift-network-diagnostics/network-check-target-mfn6w","openshift-ovn-kubernetes/ovnkube-node-dzb65","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx","openshift-network-operator/iptables-alerter-q6grq","kube-system/kube-apiserver-proxy-ip-10-0-134-224.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal","openshift-multus/multus-m692r","openshift-multus/network-metrics-daemon-tkgd9"] Nov 04 04:45:59.482046 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.482024 4395 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 04 04:45:59.488564 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.488542 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-cni-bin\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.488644 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.488569 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3e3bf36f-7156-4151-9d6d-0937169dbcca-kubelet-config\") pod \"global-pull-secret-syncer-mwgm5\" (UID: \"3e3bf36f-7156-4151-9d6d-0937169dbcca\") " pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:59.488644 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.488600 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3e3bf36f-7156-4151-9d6d-0937169dbcca-original-pull-secret\") pod \"global-pull-secret-syncer-mwgm5\" (UID: \"3e3bf36f-7156-4151-9d6d-0937169dbcca\") " pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:59.488644 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.488616 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zszmj\" (UniqueName: \"kubernetes.io/projected/cab63986-8b41-42c4-b7b0-4438e54fa424-kube-api-access-zszmj\") pod \"iptables-alerter-q6grq\" (UID: \"cab63986-8b41-42c4-b7b0-4438e54fa424\") " pod="openshift-network-operator/iptables-alerter-q6grq" Nov 04 04:45:59.488644 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.488633 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-host-var-lib-cni-bin\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.488842 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.488697 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-run-openvswitch\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.488842 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.488736 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-sysctl-d\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:59.488842 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.488789 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-systemd\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:59.488944 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.488874 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp26f\" (UniqueName: \"kubernetes.io/projected/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-kube-api-access-kp26f\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:59.488992 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.488969 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a7936867-0a26-4c2e-b94d-a1756cd14873-konnectivity-ca\") pod \"konnectivity-agent-ct2mt\" (UID: \"a7936867-0a26-4c2e-b94d-a1756cd14873\") " pod="kube-system/konnectivity-agent-ct2mt" Nov 04 04:45:59.489063 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.489046 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-sys-fs\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:59.489106 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.489080 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3e3bf36f-7156-4151-9d6d-0937169dbcca-dbus\") pod \"global-pull-secret-syncer-mwgm5\" (UID: \"3e3bf36f-7156-4151-9d6d-0937169dbcca\") " pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:59.489106 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.489098 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-host-var-lib-cni-multus\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.489189 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.489138 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0c260a1-2a73-4b2b-b6a7-2f9717941a1b-metrics-certs\") pod \"network-metrics-daemon-tkgd9\" (UID: \"f0c260a1-2a73-4b2b-b6a7-2f9717941a1b\") " pod="openshift-multus/network-metrics-daemon-tkgd9" Nov 04 04:45:59.489232 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.489202 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/31e07d9c-93bc-4e8d-909b-e65406ff62df-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p9jxj\" (UID: \"31e07d9c-93bc-4e8d-909b-e65406ff62df\") " pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:59.489232 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.489220 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-run-netns\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.489345 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.489235 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-sys\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:59.489345 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.489267 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-host\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:59.489345 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.489331 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7vxq\" (UniqueName: \"kubernetes.io/projected/52f02c25-48ed-46dc-82bc-23016ffa595d-kube-api-access-b7vxq\") pod \"node-resolver-7f22t\" (UID: \"52f02c25-48ed-46dc-82bc-23016ffa595d\") " pod="openshift-dns/node-resolver-7f22t" Nov 04 04:45:59.489473 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.489383 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c1153d6-83db-4eb7-95e0-78d09bd61d6d-host\") pod \"node-ca-7tlqm\" (UID: \"5c1153d6-83db-4eb7-95e0-78d09bd61d6d\") " pod="openshift-image-registry/node-ca-7tlqm" Nov 04 04:45:59.489522 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.489477 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5c1153d6-83db-4eb7-95e0-78d09bd61d6d-serviceca\") pod \"node-ca-7tlqm\" (UID: \"5c1153d6-83db-4eb7-95e0-78d09bd61d6d\") " pod="openshift-image-registry/node-ca-7tlqm" Nov 04 04:45:59.489588 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.489568 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/31e07d9c-93bc-4e8d-909b-e65406ff62df-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p9jxj\" (UID: \"31e07d9c-93bc-4e8d-909b-e65406ff62df\") " pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:59.489638 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.489605 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gchdh\" (UniqueName: \"kubernetes.io/projected/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-kube-api-access-gchdh\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:59.489685 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.489633 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-sysconfig\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:59.489742 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.489689 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-multus-cni-dir\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.489742 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.489715 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-hostroot\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.489813 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.489741 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-multus-daemon-config\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.489813 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.489773 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-var-lib-openvswitch\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.489885 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.489819 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-run-ovn\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.489885 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.489865 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-node-log\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.489949 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.489900 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-socket-dir\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:59.489949 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.489917 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-etc-selinux\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:59.490009 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.489964 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-sysctl-conf\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:59.490009 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.489988 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-cnibin\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.490009 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.490004 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-host-run-k8s-cni-cncf-io\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.490105 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.490037 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lppk7\" (UniqueName: \"kubernetes.io/projected/31e07d9c-93bc-4e8d-909b-e65406ff62df-kube-api-access-lppk7\") pod \"multus-additional-cni-plugins-p9jxj\" (UID: \"31e07d9c-93bc-4e8d-909b-e65406ff62df\") " pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:59.490105 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.490055 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7380461b-403d-4470-8c1c-6433616b1dd2-ovn-node-metrics-cert\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.490105 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.490075 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-kubernetes\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:59.490105 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.490091 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-lib-modules\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:59.490229 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.490129 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-tuned\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:59.490229 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.490193 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-tuned\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:59.490229 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.490198 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-host-var-lib-kubelet\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.490334 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.490234 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f02c25-48ed-46dc-82bc-23016ffa595d-tmp-dir\") pod \"node-resolver-7f22t\" (UID: \"52f02c25-48ed-46dc-82bc-23016ffa595d\") " pod="openshift-dns/node-resolver-7f22t" Nov 04 04:45:59.490334 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.490265 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/31e07d9c-93bc-4e8d-909b-e65406ff62df-cni-binary-copy\") pod \"multus-additional-cni-plugins-p9jxj\" (UID: \"31e07d9c-93bc-4e8d-909b-e65406ff62df\") " pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:59.490396 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.490334 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f02c25-48ed-46dc-82bc-23016ffa595d-tmp-dir\") pod \"node-resolver-7f22t\" (UID: \"52f02c25-48ed-46dc-82bc-23016ffa595d\") " pod="openshift-dns/node-resolver-7f22t" Nov 04 04:45:59.490396 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.490361 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-systemd-units\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.490499 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.490408 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:59.490499 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.490429 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-var-lib-kubelet\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:59.490499 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.490459 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-multus-conf-dir\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.490499 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.490484 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6kb6\" (UniqueName: \"kubernetes.io/projected/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-kube-api-access-w6kb6\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.490614 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.490515 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-tmp\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:59.490614 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.490592 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-tmp\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:59.490614 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.490593 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-kubelet\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.490768 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.490624 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.491012 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.490990 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwmfd\" (UniqueName: \"kubernetes.io/projected/7380461b-403d-4470-8c1c-6433616b1dd2-kube-api-access-pwmfd\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.491084 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.491041 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-registration-dir\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:59.491084 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.491077 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7jdp\" (UniqueName: \"kubernetes.io/projected/59d18119-72a5-410f-be97-90613af651bd-kube-api-access-b7jdp\") pod \"network-check-target-mfn6w\" (UID: \"59d18119-72a5-410f-be97-90613af651bd\") " pod="openshift-network-diagnostics/network-check-target-mfn6w" Nov 04 04:45:59.491210 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.491112 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-system-cni-dir\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.491210 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.491137 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-multus-socket-dir-parent\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.491210 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.491169 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-host-run-netns\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.491684 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.491200 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-etc-kubernetes\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.491750 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.491719 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-run\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:59.491809 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.491775 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/52f02c25-48ed-46dc-82bc-23016ffa595d-hosts-file\") pod \"node-resolver-7f22t\" (UID: \"52f02c25-48ed-46dc-82bc-23016ffa595d\") " pod="openshift-dns/node-resolver-7f22t" Nov 04 04:45:59.491858 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.491821 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-cni-netd\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.491903 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.491853 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7380461b-403d-4470-8c1c-6433616b1dd2-env-overrides\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.491953 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.491900 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-cni-binary-copy\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.491953 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.491933 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31e07d9c-93bc-4e8d-909b-e65406ff62df-system-cni-dir\") pod \"multus-additional-cni-plugins-p9jxj\" (UID: \"31e07d9c-93bc-4e8d-909b-e65406ff62df\") " pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:59.492057 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.491981 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-log-socket\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.492057 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.492019 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-device-dir\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:59.492142 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.492061 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-host-run-multus-certs\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.492142 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.492092 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/31e07d9c-93bc-4e8d-909b-e65406ff62df-cnibin\") pod \"multus-additional-cni-plugins-p9jxj\" (UID: \"31e07d9c-93bc-4e8d-909b-e65406ff62df\") " pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:59.492142 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.492136 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cab63986-8b41-42c4-b7b0-4438e54fa424-iptables-alerter-script\") pod \"iptables-alerter-q6grq\" (UID: \"cab63986-8b41-42c4-b7b0-4438e54fa424\") " pod="openshift-network-operator/iptables-alerter-q6grq" Nov 04 04:45:59.492335 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.492167 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cab63986-8b41-42c4-b7b0-4438e54fa424-host-slash\") pod \"iptables-alerter-q6grq\" (UID: \"cab63986-8b41-42c4-b7b0-4438e54fa424\") " pod="openshift-network-operator/iptables-alerter-q6grq" Nov 04 04:45:59.492335 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.492204 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd5qw\" (UniqueName: \"kubernetes.io/projected/f0c260a1-2a73-4b2b-b6a7-2f9717941a1b-kube-api-access-pd5qw\") pod \"network-metrics-daemon-tkgd9\" (UID: \"f0c260a1-2a73-4b2b-b6a7-2f9717941a1b\") " pod="openshift-multus/network-metrics-daemon-tkgd9" Nov 04 04:45:59.492335 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.492235 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a7936867-0a26-4c2e-b94d-a1756cd14873-agent-certs\") pod \"konnectivity-agent-ct2mt\" (UID: \"a7936867-0a26-4c2e-b94d-a1756cd14873\") " pod="kube-system/konnectivity-agent-ct2mt" Nov 04 04:45:59.492335 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.492273 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-run-ovn-kubernetes\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.492335 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.492323 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d9jw\" (UniqueName: \"kubernetes.io/projected/5c1153d6-83db-4eb7-95e0-78d09bd61d6d-kube-api-access-5d9jw\") pod \"node-ca-7tlqm\" (UID: \"5c1153d6-83db-4eb7-95e0-78d09bd61d6d\") " pod="openshift-image-registry/node-ca-7tlqm" Nov 04 04:45:59.492543 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.492353 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/31e07d9c-93bc-4e8d-909b-e65406ff62df-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p9jxj\" (UID: \"31e07d9c-93bc-4e8d-909b-e65406ff62df\") " pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:59.492543 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.492389 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-run-systemd\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.492543 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.492426 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-etc-openvswitch\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.492543 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.492452 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7380461b-403d-4470-8c1c-6433616b1dd2-ovnkube-config\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.492543 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.492494 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7380461b-403d-4470-8c1c-6433616b1dd2-ovnkube-script-lib\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.492543 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.492524 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-modprobe-d\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:59.492786 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.492565 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-slash\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.502226 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.502207 4395 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 04:45:59.502315 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:45:59.502247 4395 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-224.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:59.502674 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.502659 4395 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 04:45:59.502733 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:45:59.502696 4395 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.us-west-2.compute.internal" Nov 04 04:45:59.593466 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593442 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-etc-openvswitch\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.593537 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593469 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-modprobe-d\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:59.593537 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593484 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-slash\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.593537 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593499 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-cni-bin\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.593537 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593514 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3e3bf36f-7156-4151-9d6d-0937169dbcca-kubelet-config\") pod \"global-pull-secret-syncer-mwgm5\" (UID: \"3e3bf36f-7156-4151-9d6d-0937169dbcca\") " pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:59.593537 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593529 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-host-var-lib-cni-bin\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.593778 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593544 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-run-openvswitch\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.593778 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593564 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-slash\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.593778 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593570 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-etc-openvswitch\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.593778 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593592 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-run-openvswitch\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.593778 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593597 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3e3bf36f-7156-4151-9d6d-0937169dbcca-kubelet-config\") pod \"global-pull-secret-syncer-mwgm5\" (UID: \"3e3bf36f-7156-4151-9d6d-0937169dbcca\") " pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:59.593778 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593603 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-host-var-lib-cni-bin\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.593778 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593613 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-modprobe-d\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:59.593778 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593629 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-sysctl-d\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:59.593778 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593639 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-cni-bin\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.593778 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593647 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-systemd\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:59.593778 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593681 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-systemd\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:59.593778 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593696 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-sys-fs\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:59.593778 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593719 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-sysctl-d\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:59.593778 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593723 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3e3bf36f-7156-4151-9d6d-0937169dbcca-dbus\") pod \"global-pull-secret-syncer-mwgm5\" (UID: \"3e3bf36f-7156-4151-9d6d-0937169dbcca\") " pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:59.593778 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593758 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-host-var-lib-cni-multus\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.593778 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593766 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-sys-fs\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:59.593778 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593775 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/31e07d9c-93bc-4e8d-909b-e65406ff62df-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p9jxj\" (UID: \"31e07d9c-93bc-4e8d-909b-e65406ff62df\") " pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:59.593778 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593783 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3e3bf36f-7156-4151-9d6d-0937169dbcca-dbus\") pod \"global-pull-secret-syncer-mwgm5\" (UID: \"3e3bf36f-7156-4151-9d6d-0937169dbcca\") " pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593801 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-run-netns\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593809 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-host-var-lib-cni-multus\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593827 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-sys\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593830 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/31e07d9c-93bc-4e8d-909b-e65406ff62df-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p9jxj\" (UID: \"31e07d9c-93bc-4e8d-909b-e65406ff62df\") " pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593855 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-run-netns\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593861 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-host\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593875 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-sys\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593892 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c1153d6-83db-4eb7-95e0-78d09bd61d6d-host\") pod \"node-ca-7tlqm\" (UID: \"5c1153d6-83db-4eb7-95e0-78d09bd61d6d\") " pod="openshift-image-registry/node-ca-7tlqm" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593908 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-host\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593911 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-sysconfig\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593935 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-sysconfig\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593943 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-multus-cni-dir\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593967 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-hostroot\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593945 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c1153d6-83db-4eb7-95e0-78d09bd61d6d-host\") pod \"node-ca-7tlqm\" (UID: \"5c1153d6-83db-4eb7-95e0-78d09bd61d6d\") " pod="openshift-image-registry/node-ca-7tlqm" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593986 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-multus-cni-dir\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.593991 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-var-lib-openvswitch\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594012 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-hostroot\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594022 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-var-lib-openvswitch\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594033 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-run-ovn\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594057 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-node-log\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594079 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-socket-dir\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594100 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-run-ovn\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594103 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-etc-selinux\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594127 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-sysctl-conf\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594136 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-socket-dir\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594106 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-node-log\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594139 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-etc-selinux\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594151 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-cnibin\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594180 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-cnibin\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594181 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-host-run-k8s-cni-cncf-io\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594220 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-kubernetes\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594240 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-sysctl-conf\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594243 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-lib-modules\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594276 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-host-run-k8s-cni-cncf-io\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594285 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-etc-kubernetes\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594274 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-host-var-lib-kubelet\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594331 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-systemd-units\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594337 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-host-var-lib-kubelet\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594346 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594359 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-lib-modules\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594361 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-systemd-units\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594362 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-var-lib-kubelet\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594405 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-var-lib-kubelet\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594411 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594408 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-multus-conf-dir\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594425 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-multus-conf-dir\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594443 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-kubelet\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594465 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594483 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-registration-dir\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594512 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-system-cni-dir\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594525 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594536 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-multus-socket-dir-parent\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594561 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-host-run-netns\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594572 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-registration-dir\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594536 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-kubelet\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.594563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594582 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-multus-socket-dir-parent\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.596065 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594579 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-system-cni-dir\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.596065 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594603 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-etc-kubernetes\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.596065 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594627 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-etc-kubernetes\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.596065 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594631 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-run\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:59.596065 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594642 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-host-run-netns\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.596065 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594673 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-run\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:59.596065 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594655 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/52f02c25-48ed-46dc-82bc-23016ffa595d-hosts-file\") pod \"node-resolver-7f22t\" (UID: \"52f02c25-48ed-46dc-82bc-23016ffa595d\") " pod="openshift-dns/node-resolver-7f22t" Nov 04 04:45:59.596065 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594695 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/52f02c25-48ed-46dc-82bc-23016ffa595d-hosts-file\") pod \"node-resolver-7f22t\" (UID: \"52f02c25-48ed-46dc-82bc-23016ffa595d\") " pod="openshift-dns/node-resolver-7f22t" Nov 04 04:45:59.596065 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594712 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-cni-netd\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.596065 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594732 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31e07d9c-93bc-4e8d-909b-e65406ff62df-system-cni-dir\") pod \"multus-additional-cni-plugins-p9jxj\" (UID: \"31e07d9c-93bc-4e8d-909b-e65406ff62df\") " pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:59.596065 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594746 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-log-socket\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.596065 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594761 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-device-dir\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:59.596065 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594784 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-host-run-multus-certs\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.596065 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594800 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-cni-netd\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.596065 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594809 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-device-dir\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:59.596065 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594809 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/31e07d9c-93bc-4e8d-909b-e65406ff62df-cnibin\") pod \"multus-additional-cni-plugins-p9jxj\" (UID: \"31e07d9c-93bc-4e8d-909b-e65406ff62df\") " pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:59.596065 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594828 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31e07d9c-93bc-4e8d-909b-e65406ff62df-system-cni-dir\") pod \"multus-additional-cni-plugins-p9jxj\" (UID: \"31e07d9c-93bc-4e8d-909b-e65406ff62df\") " pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:59.596065 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594837 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cab63986-8b41-42c4-b7b0-4438e54fa424-host-slash\") pod \"iptables-alerter-q6grq\" (UID: \"cab63986-8b41-42c4-b7b0-4438e54fa424\") " pod="openshift-network-operator/iptables-alerter-q6grq" Nov 04 04:45:59.596065 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594840 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/31e07d9c-93bc-4e8d-909b-e65406ff62df-cnibin\") pod \"multus-additional-cni-plugins-p9jxj\" (UID: \"31e07d9c-93bc-4e8d-909b-e65406ff62df\") " pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:59.596065 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594839 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-host-run-multus-certs\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.596065 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594854 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-run-ovn-kubernetes\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.596065 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594874 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-run-systemd\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.596065 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594834 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-log-socket\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.596065 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594874 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-host-run-ovn-kubernetes\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.596065 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594890 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cab63986-8b41-42c4-b7b0-4438e54fa424-host-slash\") pod \"iptables-alerter-q6grq\" (UID: \"cab63986-8b41-42c4-b7b0-4438e54fa424\") " pod="openshift-network-operator/iptables-alerter-q6grq" Nov 04 04:45:59.596065 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.594918 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7380461b-403d-4470-8c1c-6433616b1dd2-run-systemd\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.607938 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.607915 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-45k46\"" Nov 04 04:45:59.608024 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.607971 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-l2xwd\"" Nov 04 04:45:59.608134 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.608117 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 04 04:45:59.608199 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.608182 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-gwbb5\"" Nov 04 04:45:59.608244 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.608220 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 04 04:45:59.608290 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.608250 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 04 04:45:59.608290 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.608263 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8xbd4\"" Nov 04 04:45:59.608419 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.608354 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 04 04:45:59.608419 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.608359 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 04 04:45:59.608419 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.608394 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 04 04:45:59.608419 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.608406 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 04 04:45:59.608573 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.608421 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 04 04:45:59.608573 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.608435 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 04 04:45:59.608573 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.608464 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7qqph\"" Nov 04 04:45:59.608573 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.608468 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ss4dp\"" Nov 04 04:45:59.608573 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.608479 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 04 04:45:59.608573 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.608507 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 04 04:45:59.608818 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.608589 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 04 04:45:59.608818 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.608609 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 04 04:45:59.608818 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.608610 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kjpq6\"" Nov 04 04:45:59.608818 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.608627 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 04 04:45:59.608818 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.608649 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 04 04:45:59.608818 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.608656 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-77ccl\"" Nov 04 04:45:59.608818 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.608660 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 04 04:45:59.608818 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.608661 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 04 04:45:59.608818 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.608674 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 04 04:45:59.608818 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.608664 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 04 04:45:59.608818 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.608687 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 04 04:45:59.608818 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.608708 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 04 04:45:59.608818 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.608721 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 04 04:45:59.608818 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.608763 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-nhf59\"" Nov 04 04:45:59.608818 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.608778 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rrdk2\"" Nov 04 04:45:59.608818 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.608787 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-m97m8\"" Nov 04 04:45:59.608818 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.608809 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 04 04:45:59.608818 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.608812 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 04 04:45:59.608818 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.608822 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-95fm7\"" Nov 04 04:45:59.608818 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.608795 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 04 04:45:59.609606 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.608837 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 04 04:45:59.609606 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.608842 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 04 04:45:59.609606 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.608881 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 04 04:45:59.609606 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.608925 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 04 04:45:59.609606 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.609123 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3e3bf36f-7156-4151-9d6d-0937169dbcca-original-pull-secret\") pod \"global-pull-secret-syncer-mwgm5\" (UID: \"3e3bf36f-7156-4151-9d6d-0937169dbcca\") " pod="kube-system/global-pull-secret-syncer-mwgm5" Nov 04 04:45:59.609606 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.609412 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a7936867-0a26-4c2e-b94d-a1756cd14873-konnectivity-ca\") pod \"konnectivity-agent-ct2mt\" (UID: \"a7936867-0a26-4c2e-b94d-a1756cd14873\") " pod="kube-system/konnectivity-agent-ct2mt" Nov 04 04:45:59.610060 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.610041 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/31e07d9c-93bc-4e8d-909b-e65406ff62df-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p9jxj\" (UID: \"31e07d9c-93bc-4e8d-909b-e65406ff62df\") " pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:59.610122 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.610045 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-multus-daemon-config\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.610291 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.610272 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5c1153d6-83db-4eb7-95e0-78d09bd61d6d-serviceca\") pod \"node-ca-7tlqm\" (UID: \"5c1153d6-83db-4eb7-95e0-78d09bd61d6d\") " pod="openshift-image-registry/node-ca-7tlqm" Nov 04 04:45:59.610524 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.610508 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/31e07d9c-93bc-4e8d-909b-e65406ff62df-cni-binary-copy\") pod \"multus-additional-cni-plugins-p9jxj\" (UID: \"31e07d9c-93bc-4e8d-909b-e65406ff62df\") " pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:59.610573 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.610552 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7380461b-403d-4470-8c1c-6433616b1dd2-ovn-node-metrics-cert\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.612517 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.612502 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 04 04:45:59.612562 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.612541 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 04 04:45:59.613686 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.613665 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a7936867-0a26-4c2e-b94d-a1756cd14873-agent-certs\") pod \"konnectivity-agent-ct2mt\" (UID: \"a7936867-0a26-4c2e-b94d-a1756cd14873\") " pod="kube-system/konnectivity-agent-ct2mt" Nov 04 04:45:59.613743 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.613675 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7380461b-403d-4470-8c1c-6433616b1dd2-env-overrides\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.613743 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.613720 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-cni-binary-copy\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.613743 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.613733 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cab63986-8b41-42c4-b7b0-4438e54fa424-iptables-alerter-script\") pod \"iptables-alerter-q6grq\" (UID: \"cab63986-8b41-42c4-b7b0-4438e54fa424\") " pod="openshift-network-operator/iptables-alerter-q6grq" Nov 04 04:45:59.613833 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.613780 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/31e07d9c-93bc-4e8d-909b-e65406ff62df-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p9jxj\" (UID: \"31e07d9c-93bc-4e8d-909b-e65406ff62df\") " pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:59.613914 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.613902 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7380461b-403d-4470-8c1c-6433616b1dd2-ovnkube-config\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.614015 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.613994 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7380461b-403d-4470-8c1c-6433616b1dd2-ovnkube-script-lib\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.616190 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.616162 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6kb6\" (UniqueName: \"kubernetes.io/projected/bc352ca6-4a2d-4f7f-a434-7c45dd704e4b-kube-api-access-w6kb6\") pod \"multus-m692r\" (UID: \"bc352ca6-4a2d-4f7f-a434-7c45dd704e4b\") " pod="openshift-multus/multus-m692r" Nov 04 04:45:59.616280 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.616228 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zszmj\" (UniqueName: \"kubernetes.io/projected/cab63986-8b41-42c4-b7b0-4438e54fa424-kube-api-access-zszmj\") pod \"iptables-alerter-q6grq\" (UID: \"cab63986-8b41-42c4-b7b0-4438e54fa424\") " pod="openshift-network-operator/iptables-alerter-q6grq" Nov 04 04:45:59.616280 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.616255 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp26f\" (UniqueName: \"kubernetes.io/projected/9ae0b6cf-e85a-40a1-b61b-01bd5fde3643-kube-api-access-kp26f\") pod \"tuned-dj7mc\" (UID: \"9ae0b6cf-e85a-40a1-b61b-01bd5fde3643\") " pod="openshift-cluster-node-tuning-operator/tuned-dj7mc" Nov 04 04:45:59.616434 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.616381 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd5qw\" (UniqueName: \"kubernetes.io/projected/f0c260a1-2a73-4b2b-b6a7-2f9717941a1b-kube-api-access-pd5qw\") pod \"network-metrics-daemon-tkgd9\" (UID: \"f0c260a1-2a73-4b2b-b6a7-2f9717941a1b\") " pod="openshift-multus/network-metrics-daemon-tkgd9" Nov 04 04:45:59.616434 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.616426 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gchdh\" (UniqueName: \"kubernetes.io/projected/7422e4c1-59b2-4bdc-8fb1-7542ba692a6a-kube-api-access-gchdh\") pod \"aws-ebs-csi-driver-node-xbtvx\" (UID: \"7422e4c1-59b2-4bdc-8fb1-7542ba692a6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbtvx" Nov 04 04:45:59.616646 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.616627 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7vxq\" (UniqueName: \"kubernetes.io/projected/52f02c25-48ed-46dc-82bc-23016ffa595d-kube-api-access-b7vxq\") pod \"node-resolver-7f22t\" (UID: \"52f02c25-48ed-46dc-82bc-23016ffa595d\") " pod="openshift-dns/node-resolver-7f22t" Nov 04 04:45:59.616710 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.616690 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lppk7\" (UniqueName: \"kubernetes.io/projected/31e07d9c-93bc-4e8d-909b-e65406ff62df-kube-api-access-lppk7\") pod \"multus-additional-cni-plugins-p9jxj\" (UID: \"31e07d9c-93bc-4e8d-909b-e65406ff62df\") " pod="openshift-multus/multus-additional-cni-plugins-p9jxj" Nov 04 04:45:59.616710 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.616700 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwmfd\" (UniqueName: \"kubernetes.io/projected/7380461b-403d-4470-8c1c-6433616b1dd2-kube-api-access-pwmfd\") pod \"ovnkube-node-dzb65\" (UID: \"7380461b-403d-4470-8c1c-6433616b1dd2\") " pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:45:59.616868 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.616853 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7jdp\" (UniqueName: \"kubernetes.io/projected/59d18119-72a5-410f-be97-90613af651bd-kube-api-access-b7jdp\") pod \"network-check-target-mfn6w\" (UID: \"59d18119-72a5-410f-be97-90613af651bd\") " pod="openshift-network-diagnostics/network-check-target-mfn6w" Nov 04 04:45:59.616936 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.616923 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d9jw\" (UniqueName: \"kubernetes.io/projected/5c1153d6-83db-4eb7-95e0-78d09bd61d6d-kube-api-access-5d9jw\") pod \"node-ca-7tlqm\" (UID: \"5c1153d6-83db-4eb7-95e0-78d09bd61d6d\") " pod="openshift-image-registry/node-ca-7tlqm" Nov 04 04:45:59.620388 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:45:59.620354 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0c260a1-2a73-4b2b-b6a7-2f9717941a1b-metrics-certs\") pod \"network-metrics-daemon-tkgd9\" (UID: \"f0c260a1-2a73-4b2b-b6a7-2f9717941a1b\") " pod="openshift-multus/network-metrics-daemon-tkgd9" Nov 04 04:46:01.183919 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:01.183877 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:46:01.184450 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:01.184429 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:46:01.448270 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:01.448203 4395 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 04:46:06.127035 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:06.126986 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mfn6w" Nov 04 04:46:09.055700 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.055664 4395 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-224.us-west-2.compute.internal" event="NodeReady" Nov 04 04:46:09.210610 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.210580 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hs4hb"] Nov 04 04:46:09.214408 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.214387 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qn6tv"] Nov 04 04:46:09.214525 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.214503 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hs4hb" Nov 04 04:46:09.216469 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.216449 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qn6tv" Nov 04 04:46:09.245648 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.245627 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5380e572-69b0-47b2-b23f-a61d13d04c7e-tmp-dir\") pod \"dns-default-hs4hb\" (UID: \"5380e572-69b0-47b2-b23f-a61d13d04c7e\") " pod="openshift-dns/dns-default-hs4hb" Nov 04 04:46:09.245761 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.245657 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2wbh\" (UniqueName: \"kubernetes.io/projected/1c4ab68b-bca9-47c4-b866-3cec70aed130-kube-api-access-f2wbh\") pod \"insights-runtime-extractor-qn6tv\" (UID: \"1c4ab68b-bca9-47c4-b866-3cec70aed130\") " pod="openshift-insights/insights-runtime-extractor-qn6tv" Nov 04 04:46:09.245761 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.245683 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c4ab68b-bca9-47c4-b866-3cec70aed130-data-volume\") pod \"insights-runtime-extractor-qn6tv\" (UID: \"1c4ab68b-bca9-47c4-b866-3cec70aed130\") " pod="openshift-insights/insights-runtime-extractor-qn6tv" Nov 04 04:46:09.245832 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.245759 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c4ab68b-bca9-47c4-b866-3cec70aed130-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qn6tv\" (UID: \"1c4ab68b-bca9-47c4-b866-3cec70aed130\") " pod="openshift-insights/insights-runtime-extractor-qn6tv" Nov 04 04:46:09.245866 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.245832 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5380e572-69b0-47b2-b23f-a61d13d04c7e-config-volume\") pod \"dns-default-hs4hb\" (UID: \"5380e572-69b0-47b2-b23f-a61d13d04c7e\") " pod="openshift-dns/dns-default-hs4hb" Nov 04 04:46:09.245899 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.245864 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c4ab68b-bca9-47c4-b866-3cec70aed130-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qn6tv\" (UID: \"1c4ab68b-bca9-47c4-b866-3cec70aed130\") " pod="openshift-insights/insights-runtime-extractor-qn6tv" Nov 04 04:46:09.245899 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.245883 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c4ab68b-bca9-47c4-b866-3cec70aed130-crio-socket\") pod \"insights-runtime-extractor-qn6tv\" (UID: \"1c4ab68b-bca9-47c4-b866-3cec70aed130\") " pod="openshift-insights/insights-runtime-extractor-qn6tv" Nov 04 04:46:09.245961 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.245934 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gr78\" (UniqueName: \"kubernetes.io/projected/5380e572-69b0-47b2-b23f-a61d13d04c7e-kube-api-access-6gr78\") pod \"dns-default-hs4hb\" (UID: \"5380e572-69b0-47b2-b23f-a61d13d04c7e\") " pod="openshift-dns/dns-default-hs4hb" Nov 04 04:46:09.245993 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.245962 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5380e572-69b0-47b2-b23f-a61d13d04c7e-metrics-tls\") pod \"dns-default-hs4hb\" (UID: \"5380e572-69b0-47b2-b23f-a61d13d04c7e\") " pod="openshift-dns/dns-default-hs4hb" Nov 04 04:46:09.257003 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.256983 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tch2k"] Nov 04 04:46:09.258839 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.258825 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tch2k" Nov 04 04:46:09.264104 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.264054 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 04 04:46:09.264468 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.264441 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jn85r\"" Nov 04 04:46:09.264468 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.264456 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 04 04:46:09.264967 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.264949 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 04 04:46:09.265046 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.265028 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 04 04:46:09.265355 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.265339 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 04 04:46:09.265423 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.265405 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-frtvg\"" Nov 04 04:46:09.265486 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.265467 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 04 04:46:09.280992 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.280969 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hs4hb"] Nov 04 04:46:09.303680 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.303655 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qn6tv"] Nov 04 04:46:09.330709 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.330657 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tch2k"] Nov 04 04:46:09.346610 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.346592 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c4ab68b-bca9-47c4-b866-3cec70aed130-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qn6tv\" (UID: \"1c4ab68b-bca9-47c4-b866-3cec70aed130\") " pod="openshift-insights/insights-runtime-extractor-qn6tv" Nov 04 04:46:09.346688 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.346622 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b74d8f35-5881-448b-a0f7-7373f258c742-cert\") pod \"ingress-canary-tch2k\" (UID: \"b74d8f35-5881-448b-a0f7-7373f258c742\") " pod="openshift-ingress-canary/ingress-canary-tch2k" Nov 04 04:46:09.346688 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.346645 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5380e572-69b0-47b2-b23f-a61d13d04c7e-config-volume\") pod \"dns-default-hs4hb\" (UID: \"5380e572-69b0-47b2-b23f-a61d13d04c7e\") " pod="openshift-dns/dns-default-hs4hb" Nov 04 04:46:09.346688 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.346671 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c4ab68b-bca9-47c4-b866-3cec70aed130-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qn6tv\" (UID: \"1c4ab68b-bca9-47c4-b866-3cec70aed130\") " pod="openshift-insights/insights-runtime-extractor-qn6tv" Nov 04 04:46:09.346802 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.346788 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c4ab68b-bca9-47c4-b866-3cec70aed130-crio-socket\") pod \"insights-runtime-extractor-qn6tv\" (UID: \"1c4ab68b-bca9-47c4-b866-3cec70aed130\") " pod="openshift-insights/insights-runtime-extractor-qn6tv" Nov 04 04:46:09.346841 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.346815 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gr78\" (UniqueName: \"kubernetes.io/projected/5380e572-69b0-47b2-b23f-a61d13d04c7e-kube-api-access-6gr78\") pod \"dns-default-hs4hb\" (UID: \"5380e572-69b0-47b2-b23f-a61d13d04c7e\") " pod="openshift-dns/dns-default-hs4hb" Nov 04 04:46:09.346883 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.346839 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f55wf\" (UniqueName: \"kubernetes.io/projected/b74d8f35-5881-448b-a0f7-7373f258c742-kube-api-access-f55wf\") pod \"ingress-canary-tch2k\" (UID: \"b74d8f35-5881-448b-a0f7-7373f258c742\") " pod="openshift-ingress-canary/ingress-canary-tch2k" Nov 04 04:46:09.346883 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.346874 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5380e572-69b0-47b2-b23f-a61d13d04c7e-metrics-tls\") pod \"dns-default-hs4hb\" (UID: \"5380e572-69b0-47b2-b23f-a61d13d04c7e\") " pod="openshift-dns/dns-default-hs4hb" Nov 04 04:46:09.346962 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.346896 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5380e572-69b0-47b2-b23f-a61d13d04c7e-tmp-dir\") pod \"dns-default-hs4hb\" (UID: \"5380e572-69b0-47b2-b23f-a61d13d04c7e\") " pod="openshift-dns/dns-default-hs4hb" Nov 04 04:46:09.346962 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.346920 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c4ab68b-bca9-47c4-b866-3cec70aed130-crio-socket\") pod \"insights-runtime-extractor-qn6tv\" (UID: \"1c4ab68b-bca9-47c4-b866-3cec70aed130\") " pod="openshift-insights/insights-runtime-extractor-qn6tv" Nov 04 04:46:09.347039 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.346978 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2wbh\" (UniqueName: \"kubernetes.io/projected/1c4ab68b-bca9-47c4-b866-3cec70aed130-kube-api-access-f2wbh\") pod \"insights-runtime-extractor-qn6tv\" (UID: \"1c4ab68b-bca9-47c4-b866-3cec70aed130\") " pod="openshift-insights/insights-runtime-extractor-qn6tv" Nov 04 04:46:09.347039 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.347025 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c4ab68b-bca9-47c4-b866-3cec70aed130-data-volume\") pod \"insights-runtime-extractor-qn6tv\" (UID: \"1c4ab68b-bca9-47c4-b866-3cec70aed130\") " pod="openshift-insights/insights-runtime-extractor-qn6tv" Nov 04 04:46:09.347145 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.347073 4395 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 04 04:46:09.347199 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.347148 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5380e572-69b0-47b2-b23f-a61d13d04c7e-tmp-dir\") pod \"dns-default-hs4hb\" (UID: \"5380e572-69b0-47b2-b23f-a61d13d04c7e\") " pod="openshift-dns/dns-default-hs4hb" Nov 04 04:46:09.347376 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.347357 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5380e572-69b0-47b2-b23f-a61d13d04c7e-config-volume\") pod \"dns-default-hs4hb\" (UID: \"5380e572-69b0-47b2-b23f-a61d13d04c7e\") " pod="openshift-dns/dns-default-hs4hb" Nov 04 04:46:09.347456 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.347363 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c4ab68b-bca9-47c4-b866-3cec70aed130-data-volume\") pod \"insights-runtime-extractor-qn6tv\" (UID: \"1c4ab68b-bca9-47c4-b866-3cec70aed130\") " pod="openshift-insights/insights-runtime-extractor-qn6tv" Nov 04 04:46:09.347563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.347545 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c4ab68b-bca9-47c4-b866-3cec70aed130-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qn6tv\" (UID: \"1c4ab68b-bca9-47c4-b866-3cec70aed130\") " pod="openshift-insights/insights-runtime-extractor-qn6tv" Nov 04 04:46:09.349185 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.349169 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c4ab68b-bca9-47c4-b866-3cec70aed130-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qn6tv\" (UID: \"1c4ab68b-bca9-47c4-b866-3cec70aed130\") " pod="openshift-insights/insights-runtime-extractor-qn6tv" Nov 04 04:46:09.349259 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.349198 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5380e572-69b0-47b2-b23f-a61d13d04c7e-metrics-tls\") pod \"dns-default-hs4hb\" (UID: \"5380e572-69b0-47b2-b23f-a61d13d04c7e\") " pod="openshift-dns/dns-default-hs4hb" Nov 04 04:46:09.364352 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.364334 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 04 04:46:09.364440 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.364354 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 04 04:46:09.364440 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.364373 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 04 04:46:09.365063 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.365043 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-4k9ql\"" Nov 04 04:46:09.405109 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.405087 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gr78\" (UniqueName: \"kubernetes.io/projected/5380e572-69b0-47b2-b23f-a61d13d04c7e-kube-api-access-6gr78\") pod \"dns-default-hs4hb\" (UID: \"5380e572-69b0-47b2-b23f-a61d13d04c7e\") " pod="openshift-dns/dns-default-hs4hb" Nov 04 04:46:09.405189 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.405120 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2wbh\" (UniqueName: \"kubernetes.io/projected/1c4ab68b-bca9-47c4-b866-3cec70aed130-kube-api-access-f2wbh\") pod \"insights-runtime-extractor-qn6tv\" (UID: \"1c4ab68b-bca9-47c4-b866-3cec70aed130\") " pod="openshift-insights/insights-runtime-extractor-qn6tv" Nov 04 04:46:09.447923 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.447903 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f55wf\" (UniqueName: \"kubernetes.io/projected/b74d8f35-5881-448b-a0f7-7373f258c742-kube-api-access-f55wf\") pod \"ingress-canary-tch2k\" (UID: \"b74d8f35-5881-448b-a0f7-7373f258c742\") " pod="openshift-ingress-canary/ingress-canary-tch2k" Nov 04 04:46:09.448007 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.447940 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b74d8f35-5881-448b-a0f7-7373f258c742-cert\") pod \"ingress-canary-tch2k\" (UID: \"b74d8f35-5881-448b-a0f7-7373f258c742\") " pod="openshift-ingress-canary/ingress-canary-tch2k" Nov 04 04:46:09.449991 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.449971 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b74d8f35-5881-448b-a0f7-7373f258c742-cert\") pod \"ingress-canary-tch2k\" (UID: \"b74d8f35-5881-448b-a0f7-7373f258c742\") " pod="openshift-ingress-canary/ingress-canary-tch2k" Nov 04 04:46:09.505886 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.505864 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f55wf\" (UniqueName: \"kubernetes.io/projected/b74d8f35-5881-448b-a0f7-7373f258c742-kube-api-access-f55wf\") pod \"ingress-canary-tch2k\" (UID: \"b74d8f35-5881-448b-a0f7-7373f258c742\") " pod="openshift-ingress-canary/ingress-canary-tch2k" Nov 04 04:46:09.529061 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.529041 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hs4hb" Nov 04 04:46:09.535016 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.534993 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qn6tv" Nov 04 04:46:09.568200 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.568157 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tch2k" Nov 04 04:46:09.897175 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.896967 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hs4hb"] Nov 04 04:46:09.901083 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:46:09.901044 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5380e572_69b0_47b2_b23f_a61d13d04c7e.slice/crio-ba3cb1205238aa80e116dda7ccdd60f4a7ff5075ddd985833ead3cffc216e066 WatchSource:0}: Error finding container ba3cb1205238aa80e116dda7ccdd60f4a7ff5075ddd985833ead3cffc216e066: Status 404 returned error can't find the container with id ba3cb1205238aa80e116dda7ccdd60f4a7ff5075ddd985833ead3cffc216e066 Nov 04 04:46:09.903028 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.903013 4395 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 04:46:09.938427 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.938404 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tch2k"] Nov 04 04:46:09.941858 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:09.941833 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qn6tv"] Nov 04 04:46:09.941949 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:46:09.941874 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb74d8f35_5881_448b_a0f7_7373f258c742.slice/crio-a3e38e86f03a340168b572517ef9b181bfab2cbdc2f7151810b647f29daaedf7 WatchSource:0}: Error finding container a3e38e86f03a340168b572517ef9b181bfab2cbdc2f7151810b647f29daaedf7: Status 404 returned error can't find the container with id a3e38e86f03a340168b572517ef9b181bfab2cbdc2f7151810b647f29daaedf7 Nov 04 04:46:09.945136 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:46:09.945112 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c4ab68b_bca9_47c4_b866_3cec70aed130.slice/crio-4c655c83cab022e6d3ac3e972c326fb9229bf490d38db879614ddb47309d8c37 WatchSource:0}: Error finding container 4c655c83cab022e6d3ac3e972c326fb9229bf490d38db879614ddb47309d8c37: Status 404 returned error can't find the container with id 4c655c83cab022e6d3ac3e972c326fb9229bf490d38db879614ddb47309d8c37 Nov 04 04:46:10.469340 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:10.469290 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tch2k" event={"ID":"b74d8f35-5881-448b-a0f7-7373f258c742","Type":"ContainerStarted","Data":"a3e38e86f03a340168b572517ef9b181bfab2cbdc2f7151810b647f29daaedf7"} Nov 04 04:46:10.470759 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:10.470734 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qn6tv" event={"ID":"1c4ab68b-bca9-47c4-b866-3cec70aed130","Type":"ContainerStarted","Data":"3943da908ce0cf5937354c059f9fda52f9cefb719f57072237cc242777142154"} Nov 04 04:46:10.470888 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:10.470784 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qn6tv" event={"ID":"1c4ab68b-bca9-47c4-b866-3cec70aed130","Type":"ContainerStarted","Data":"4c655c83cab022e6d3ac3e972c326fb9229bf490d38db879614ddb47309d8c37"} Nov 04 04:46:10.471860 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:10.471831 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hs4hb" event={"ID":"5380e572-69b0-47b2-b23f-a61d13d04c7e","Type":"ContainerStarted","Data":"ba3cb1205238aa80e116dda7ccdd60f4a7ff5075ddd985833ead3cffc216e066"} Nov 04 04:46:10.728541 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:10.728483 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-vnlrf"] Nov 04 04:46:10.730614 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:10.730598 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-vnlrf" Nov 04 04:46:10.759369 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:10.759339 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk564\" (UniqueName: \"kubernetes.io/projected/e0197f53-2bf1-4a3c-ae89-d8a74ff7c33c-kube-api-access-xk564\") pod \"prometheus-operator-9cd89b74c-vnlrf\" (UID: \"e0197f53-2bf1-4a3c-ae89-d8a74ff7c33c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vnlrf" Nov 04 04:46:10.759487 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:10.759383 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e0197f53-2bf1-4a3c-ae89-d8a74ff7c33c-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-vnlrf\" (UID: \"e0197f53-2bf1-4a3c-ae89-d8a74ff7c33c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vnlrf" Nov 04 04:46:10.759487 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:10.759418 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e0197f53-2bf1-4a3c-ae89-d8a74ff7c33c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-vnlrf\" (UID: \"e0197f53-2bf1-4a3c-ae89-d8a74ff7c33c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vnlrf" Nov 04 04:46:10.759487 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:10.759466 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0197f53-2bf1-4a3c-ae89-d8a74ff7c33c-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-vnlrf\" (UID: \"e0197f53-2bf1-4a3c-ae89-d8a74ff7c33c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vnlrf" Nov 04 04:46:10.779694 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:10.779667 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 04 04:46:10.779804 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:10.779726 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 04 04:46:10.779804 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:10.779765 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 04 04:46:10.779804 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:10.779787 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-7qlcr\"" Nov 04 04:46:10.827085 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:10.827061 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 04 04:46:10.827181 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:10.827110 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 04 04:46:10.827608 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:10.827589 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-vnlrf"] Nov 04 04:46:10.859690 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:10.859667 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0197f53-2bf1-4a3c-ae89-d8a74ff7c33c-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-vnlrf\" (UID: \"e0197f53-2bf1-4a3c-ae89-d8a74ff7c33c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vnlrf" Nov 04 04:46:10.859761 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:10.859705 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk564\" (UniqueName: \"kubernetes.io/projected/e0197f53-2bf1-4a3c-ae89-d8a74ff7c33c-kube-api-access-xk564\") pod \"prometheus-operator-9cd89b74c-vnlrf\" (UID: \"e0197f53-2bf1-4a3c-ae89-d8a74ff7c33c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vnlrf" Nov 04 04:46:10.859761 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:10.859725 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e0197f53-2bf1-4a3c-ae89-d8a74ff7c33c-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-vnlrf\" (UID: \"e0197f53-2bf1-4a3c-ae89-d8a74ff7c33c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vnlrf" Nov 04 04:46:10.859761 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:10.859749 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e0197f53-2bf1-4a3c-ae89-d8a74ff7c33c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-vnlrf\" (UID: \"e0197f53-2bf1-4a3c-ae89-d8a74ff7c33c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vnlrf" Nov 04 04:46:10.860370 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:10.860350 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0197f53-2bf1-4a3c-ae89-d8a74ff7c33c-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-vnlrf\" (UID: \"e0197f53-2bf1-4a3c-ae89-d8a74ff7c33c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vnlrf" Nov 04 04:46:10.862088 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:10.862066 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e0197f53-2bf1-4a3c-ae89-d8a74ff7c33c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-vnlrf\" (UID: \"e0197f53-2bf1-4a3c-ae89-d8a74ff7c33c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vnlrf" Nov 04 04:46:10.862160 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:10.862146 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e0197f53-2bf1-4a3c-ae89-d8a74ff7c33c-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-vnlrf\" (UID: \"e0197f53-2bf1-4a3c-ae89-d8a74ff7c33c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vnlrf" Nov 04 04:46:10.916919 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:10.916892 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk564\" (UniqueName: \"kubernetes.io/projected/e0197f53-2bf1-4a3c-ae89-d8a74ff7c33c-kube-api-access-xk564\") pod \"prometheus-operator-9cd89b74c-vnlrf\" (UID: \"e0197f53-2bf1-4a3c-ae89-d8a74ff7c33c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vnlrf" Nov 04 04:46:11.041797 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:11.041747 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-vnlrf" Nov 04 04:46:11.401063 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:11.401031 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-vnlrf"] Nov 04 04:46:11.405810 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:46:11.405781 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0197f53_2bf1_4a3c_ae89_d8a74ff7c33c.slice/crio-864cdef1a5545fca35a85653e559b80783ce2e26f1c5cb798d1dc2ea71f73931 WatchSource:0}: Error finding container 864cdef1a5545fca35a85653e559b80783ce2e26f1c5cb798d1dc2ea71f73931: Status 404 returned error can't find the container with id 864cdef1a5545fca35a85653e559b80783ce2e26f1c5cb798d1dc2ea71f73931 Nov 04 04:46:11.475103 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:11.475052 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-vnlrf" event={"ID":"e0197f53-2bf1-4a3c-ae89-d8a74ff7c33c","Type":"ContainerStarted","Data":"864cdef1a5545fca35a85653e559b80783ce2e26f1c5cb798d1dc2ea71f73931"} Nov 04 04:46:11.914628 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:11.914606 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-84d44d8fcb-hmt2s"] Nov 04 04:46:11.916485 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:11.916470 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84d44d8fcb-hmt2s" Nov 04 04:46:11.965113 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:11.965087 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 04 04:46:11.965264 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:11.965144 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 04 04:46:11.965264 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:11.965203 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 04 04:46:11.965957 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:11.965437 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 04 04:46:11.965957 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:11.965554 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 04 04:46:11.965957 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:11.965566 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7pqf8\"" Nov 04 04:46:11.967764 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:11.967740 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/abe56d25-e504-4167-8a70-95cc1918b64f-console-config\") pod \"console-84d44d8fcb-hmt2s\" (UID: \"abe56d25-e504-4167-8a70-95cc1918b64f\") " pod="openshift-console/console-84d44d8fcb-hmt2s" Nov 04 04:46:11.967851 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:11.967781 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/abe56d25-e504-4167-8a70-95cc1918b64f-oauth-serving-cert\") pod \"console-84d44d8fcb-hmt2s\" (UID: \"abe56d25-e504-4167-8a70-95cc1918b64f\") " pod="openshift-console/console-84d44d8fcb-hmt2s" Nov 04 04:46:11.967851 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:11.967807 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n675z\" (UniqueName: \"kubernetes.io/projected/abe56d25-e504-4167-8a70-95cc1918b64f-kube-api-access-n675z\") pod \"console-84d44d8fcb-hmt2s\" (UID: \"abe56d25-e504-4167-8a70-95cc1918b64f\") " pod="openshift-console/console-84d44d8fcb-hmt2s" Nov 04 04:46:11.967957 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:11.967929 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/abe56d25-e504-4167-8a70-95cc1918b64f-console-serving-cert\") pod \"console-84d44d8fcb-hmt2s\" (UID: \"abe56d25-e504-4167-8a70-95cc1918b64f\") " pod="openshift-console/console-84d44d8fcb-hmt2s" Nov 04 04:46:11.968018 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:11.967965 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/abe56d25-e504-4167-8a70-95cc1918b64f-console-oauth-config\") pod \"console-84d44d8fcb-hmt2s\" (UID: \"abe56d25-e504-4167-8a70-95cc1918b64f\") " pod="openshift-console/console-84d44d8fcb-hmt2s" Nov 04 04:46:11.968018 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:11.967991 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/abe56d25-e504-4167-8a70-95cc1918b64f-tmp\") pod \"console-84d44d8fcb-hmt2s\" (UID: \"abe56d25-e504-4167-8a70-95cc1918b64f\") " pod="openshift-console/console-84d44d8fcb-hmt2s" Nov 04 04:46:11.968088 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:11.968064 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/abe56d25-e504-4167-8a70-95cc1918b64f-service-ca\") pod \"console-84d44d8fcb-hmt2s\" (UID: \"abe56d25-e504-4167-8a70-95cc1918b64f\") " pod="openshift-console/console-84d44d8fcb-hmt2s" Nov 04 04:46:12.011748 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:12.011723 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 04 04:46:12.011748 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:12.011724 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 04 04:46:12.012099 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:12.012074 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84d44d8fcb-hmt2s"] Nov 04 04:46:12.068752 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:12.068734 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/abe56d25-e504-4167-8a70-95cc1918b64f-console-config\") pod \"console-84d44d8fcb-hmt2s\" (UID: \"abe56d25-e504-4167-8a70-95cc1918b64f\") " pod="openshift-console/console-84d44d8fcb-hmt2s" Nov 04 04:46:12.068858 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:12.068761 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/abe56d25-e504-4167-8a70-95cc1918b64f-oauth-serving-cert\") pod \"console-84d44d8fcb-hmt2s\" (UID: \"abe56d25-e504-4167-8a70-95cc1918b64f\") " pod="openshift-console/console-84d44d8fcb-hmt2s" Nov 04 04:46:12.068858 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:12.068779 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n675z\" (UniqueName: \"kubernetes.io/projected/abe56d25-e504-4167-8a70-95cc1918b64f-kube-api-access-n675z\") pod \"console-84d44d8fcb-hmt2s\" (UID: \"abe56d25-e504-4167-8a70-95cc1918b64f\") " pod="openshift-console/console-84d44d8fcb-hmt2s" Nov 04 04:46:12.068858 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:12.068798 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/abe56d25-e504-4167-8a70-95cc1918b64f-console-serving-cert\") pod \"console-84d44d8fcb-hmt2s\" (UID: \"abe56d25-e504-4167-8a70-95cc1918b64f\") " pod="openshift-console/console-84d44d8fcb-hmt2s" Nov 04 04:46:12.068858 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:12.068816 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/abe56d25-e504-4167-8a70-95cc1918b64f-console-oauth-config\") pod \"console-84d44d8fcb-hmt2s\" (UID: \"abe56d25-e504-4167-8a70-95cc1918b64f\") " pod="openshift-console/console-84d44d8fcb-hmt2s" Nov 04 04:46:12.068858 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:12.068839 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/abe56d25-e504-4167-8a70-95cc1918b64f-tmp\") pod \"console-84d44d8fcb-hmt2s\" (UID: \"abe56d25-e504-4167-8a70-95cc1918b64f\") " pod="openshift-console/console-84d44d8fcb-hmt2s" Nov 04 04:46:12.069096 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:12.069070 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/abe56d25-e504-4167-8a70-95cc1918b64f-service-ca\") pod \"console-84d44d8fcb-hmt2s\" (UID: \"abe56d25-e504-4167-8a70-95cc1918b64f\") " pod="openshift-console/console-84d44d8fcb-hmt2s" Nov 04 04:46:12.069306 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:12.069276 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/abe56d25-e504-4167-8a70-95cc1918b64f-tmp\") pod \"console-84d44d8fcb-hmt2s\" (UID: \"abe56d25-e504-4167-8a70-95cc1918b64f\") " pod="openshift-console/console-84d44d8fcb-hmt2s" Nov 04 04:46:12.069567 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:12.069547 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/abe56d25-e504-4167-8a70-95cc1918b64f-oauth-serving-cert\") pod \"console-84d44d8fcb-hmt2s\" (UID: \"abe56d25-e504-4167-8a70-95cc1918b64f\") " pod="openshift-console/console-84d44d8fcb-hmt2s" Nov 04 04:46:12.069635 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:12.069553 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/abe56d25-e504-4167-8a70-95cc1918b64f-console-config\") pod \"console-84d44d8fcb-hmt2s\" (UID: \"abe56d25-e504-4167-8a70-95cc1918b64f\") " pod="openshift-console/console-84d44d8fcb-hmt2s" Nov 04 04:46:12.069690 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:12.069679 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/abe56d25-e504-4167-8a70-95cc1918b64f-service-ca\") pod \"console-84d44d8fcb-hmt2s\" (UID: \"abe56d25-e504-4167-8a70-95cc1918b64f\") " pod="openshift-console/console-84d44d8fcb-hmt2s" Nov 04 04:46:12.071170 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:12.071145 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/abe56d25-e504-4167-8a70-95cc1918b64f-console-oauth-config\") pod \"console-84d44d8fcb-hmt2s\" (UID: \"abe56d25-e504-4167-8a70-95cc1918b64f\") " pod="openshift-console/console-84d44d8fcb-hmt2s" Nov 04 04:46:12.071247 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:12.071221 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/abe56d25-e504-4167-8a70-95cc1918b64f-console-serving-cert\") pod \"console-84d44d8fcb-hmt2s\" (UID: \"abe56d25-e504-4167-8a70-95cc1918b64f\") " pod="openshift-console/console-84d44d8fcb-hmt2s" Nov 04 04:46:12.126215 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:12.126192 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n675z\" (UniqueName: \"kubernetes.io/projected/abe56d25-e504-4167-8a70-95cc1918b64f-kube-api-access-n675z\") pod \"console-84d44d8fcb-hmt2s\" (UID: \"abe56d25-e504-4167-8a70-95cc1918b64f\") " pod="openshift-console/console-84d44d8fcb-hmt2s" Nov 04 04:46:12.226344 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:12.226277 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84d44d8fcb-hmt2s" Nov 04 04:46:12.479382 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:12.479327 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qn6tv" event={"ID":"1c4ab68b-bca9-47c4-b866-3cec70aed130","Type":"ContainerStarted","Data":"59e46bed2695a94a9c0715570f1c33a3d2a3b40aac599591bc3b857ccda90723"} Nov 04 04:46:12.586372 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:12.586344 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84d44d8fcb-hmt2s"] Nov 04 04:46:12.617604 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:46:12.617580 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabe56d25_e504_4167_8a70_95cc1918b64f.slice/crio-6c2b0131f219b4a10ec9d6d5fab8d41821198333a6c9a3216915cd7a563ed05d WatchSource:0}: Error finding container 6c2b0131f219b4a10ec9d6d5fab8d41821198333a6c9a3216915cd7a563ed05d: Status 404 returned error can't find the container with id 6c2b0131f219b4a10ec9d6d5fab8d41821198333a6c9a3216915cd7a563ed05d Nov 04 04:46:13.483855 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:13.483821 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84d44d8fcb-hmt2s" event={"ID":"abe56d25-e504-4167-8a70-95cc1918b64f","Type":"ContainerStarted","Data":"6c2b0131f219b4a10ec9d6d5fab8d41821198333a6c9a3216915cd7a563ed05d"} Nov 04 04:46:14.492751 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:14.492721 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qn6tv" event={"ID":"1c4ab68b-bca9-47c4-b866-3cec70aed130","Type":"ContainerStarted","Data":"4193ad361acc84ba19466f067f4d9032f64b5144ab9258afc8dbcd1a37645cf7"} Nov 04 04:46:14.603760 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:14.603357 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qn6tv" podStartSLOduration=1.707905726 podStartE2EDuration="5.603339588s" totalImagesPullingTime="3.895433862s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 04:46:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:46:14.602963064 +0000 UTC m=+16.293045531" watchObservedRunningTime="2026-11-04 04:46:14.603339588 +0000 UTC m=+16.293422036" Nov 04 04:46:15.498710 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:15.498576 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hs4hb" event={"ID":"5380e572-69b0-47b2-b23f-a61d13d04c7e","Type":"ContainerStarted","Data":"a830e94d7cd885e976379a70350f783bc2155537f0b42a0df2dac26bdf455df8"} Nov 04 04:46:15.498710 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:15.498642 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hs4hb" event={"ID":"5380e572-69b0-47b2-b23f-a61d13d04c7e","Type":"ContainerStarted","Data":"5799851670fa53ae07286d458b30708f657a463c8f4fe48a85e4cf2fcd6be88b"} Nov 04 04:46:15.498710 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:15.498671 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hs4hb" Nov 04 04:46:15.500879 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:15.500844 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-vnlrf" event={"ID":"e0197f53-2bf1-4a3c-ae89-d8a74ff7c33c","Type":"ContainerStarted","Data":"e52b7d0bc74375e53e882e428ff6089c15e90539f531cf07db60c1a0a7bc4ec8"} Nov 04 04:46:15.500993 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:15.500899 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-vnlrf" event={"ID":"e0197f53-2bf1-4a3c-ae89-d8a74ff7c33c","Type":"ContainerStarted","Data":"4d7def272e3601de59feeeb83b856826e038a62dc003f15d5e8c79719a1501ec"} Nov 04 04:46:15.608990 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:15.608948 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-hs4hb" podStartSLOduration=1.769565063 podStartE2EDuration="6.608936078s" totalImagesPullingTime="4.839371015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:46:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:46:15.608728557 +0000 UTC m=+17.298810982" watchObservedRunningTime="2026-11-04 04:46:15.608936078 +0000 UTC m=+17.299018522" Nov 04 04:46:15.718702 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:15.718658 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-vnlrf" podStartSLOduration=2.695975556 podStartE2EDuration="5.718645879s" totalImagesPullingTime="3.022670323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:46:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:46:15.718006321 +0000 UTC m=+17.408088768" watchObservedRunningTime="2026-11-04 04:46:15.718645879 +0000 UTC m=+17.408728323" Nov 04 04:46:15.874359 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:15.874331 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dzb65" Nov 04 04:46:17.509460 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:17.509422 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84d44d8fcb-hmt2s" event={"ID":"abe56d25-e504-4167-8a70-95cc1918b64f","Type":"ContainerStarted","Data":"bcdb44ca274c27c312bfb46acf65c7c55dcaa8202ef619983a8714cc4e5d6f2b"} Nov 04 04:46:17.620505 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:17.620461 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-84d44d8fcb-hmt2s" podStartSLOduration=2.227171983 podStartE2EDuration="6.620447793s" totalImagesPullingTime="4.39327581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:46:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:46:17.620194151 +0000 UTC m=+19.310276621" watchObservedRunningTime="2026-11-04 04:46:17.620447793 +0000 UTC m=+19.310530644" Nov 04 04:46:18.513825 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:18.513789 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tch2k" event={"ID":"b74d8f35-5881-448b-a0f7-7373f258c742","Type":"ContainerStarted","Data":"422ee6fb975088a0277027d513b87484e46d7f9b90b49c296c0970adc1034554"} Nov 04 04:46:18.621926 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:18.621879 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tch2k" podStartSLOduration=2.038779769 podStartE2EDuration="9.621866407s" totalImagesPullingTime="7.583086638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:46:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:46:18.6214399 +0000 UTC m=+20.311522362" watchObservedRunningTime="2026-11-04 04:46:18.621866407 +0000 UTC m=+20.311948854" Nov 04 04:46:18.673623 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:18.673594 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wrqmt"] Nov 04 04:46:18.676751 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:18.676732 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wrqmt" Nov 04 04:46:18.715863 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:18.715836 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b77f3812-c82c-4e4f-8cf3-571265f1629a-node-exporter-tls\") pod \"node-exporter-wrqmt\" (UID: \"b77f3812-c82c-4e4f-8cf3-571265f1629a\") " pod="openshift-monitoring/node-exporter-wrqmt" Nov 04 04:46:18.715965 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:18.715868 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b77f3812-c82c-4e4f-8cf3-571265f1629a-root\") pod \"node-exporter-wrqmt\" (UID: \"b77f3812-c82c-4e4f-8cf3-571265f1629a\") " pod="openshift-monitoring/node-exporter-wrqmt" Nov 04 04:46:18.715965 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:18.715884 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b77f3812-c82c-4e4f-8cf3-571265f1629a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wrqmt\" (UID: \"b77f3812-c82c-4e4f-8cf3-571265f1629a\") " pod="openshift-monitoring/node-exporter-wrqmt" Nov 04 04:46:18.715965 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:18.715901 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b77f3812-c82c-4e4f-8cf3-571265f1629a-node-exporter-textfile\") pod \"node-exporter-wrqmt\" (UID: \"b77f3812-c82c-4e4f-8cf3-571265f1629a\") " pod="openshift-monitoring/node-exporter-wrqmt" Nov 04 04:46:18.716071 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:18.715961 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b77f3812-c82c-4e4f-8cf3-571265f1629a-metrics-client-ca\") pod \"node-exporter-wrqmt\" (UID: \"b77f3812-c82c-4e4f-8cf3-571265f1629a\") " pod="openshift-monitoring/node-exporter-wrqmt" Nov 04 04:46:18.716071 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:18.716047 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrh47\" (UniqueName: \"kubernetes.io/projected/b77f3812-c82c-4e4f-8cf3-571265f1629a-kube-api-access-wrh47\") pod \"node-exporter-wrqmt\" (UID: \"b77f3812-c82c-4e4f-8cf3-571265f1629a\") " pod="openshift-monitoring/node-exporter-wrqmt" Nov 04 04:46:18.716130 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:18.716087 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b77f3812-c82c-4e4f-8cf3-571265f1629a-node-exporter-accelerators-collector-config\") pod \"node-exporter-wrqmt\" (UID: \"b77f3812-c82c-4e4f-8cf3-571265f1629a\") " pod="openshift-monitoring/node-exporter-wrqmt" Nov 04 04:46:18.716130 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:18.716108 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b77f3812-c82c-4e4f-8cf3-571265f1629a-node-exporter-wtmp\") pod \"node-exporter-wrqmt\" (UID: \"b77f3812-c82c-4e4f-8cf3-571265f1629a\") " pod="openshift-monitoring/node-exporter-wrqmt" Nov 04 04:46:18.716192 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:18.716154 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b77f3812-c82c-4e4f-8cf3-571265f1629a-sys\") pod \"node-exporter-wrqmt\" (UID: \"b77f3812-c82c-4e4f-8cf3-571265f1629a\") " pod="openshift-monitoring/node-exporter-wrqmt" Nov 04 04:46:18.816811 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:18.816786 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b77f3812-c82c-4e4f-8cf3-571265f1629a-node-exporter-accelerators-collector-config\") pod \"node-exporter-wrqmt\" (UID: \"b77f3812-c82c-4e4f-8cf3-571265f1629a\") " pod="openshift-monitoring/node-exporter-wrqmt" Nov 04 04:46:18.816919 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:18.816816 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b77f3812-c82c-4e4f-8cf3-571265f1629a-node-exporter-wtmp\") pod \"node-exporter-wrqmt\" (UID: \"b77f3812-c82c-4e4f-8cf3-571265f1629a\") " pod="openshift-monitoring/node-exporter-wrqmt" Nov 04 04:46:18.816919 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:18.816835 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b77f3812-c82c-4e4f-8cf3-571265f1629a-sys\") pod \"node-exporter-wrqmt\" (UID: \"b77f3812-c82c-4e4f-8cf3-571265f1629a\") " pod="openshift-monitoring/node-exporter-wrqmt" Nov 04 04:46:18.816919 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:18.816853 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b77f3812-c82c-4e4f-8cf3-571265f1629a-node-exporter-tls\") pod \"node-exporter-wrqmt\" (UID: \"b77f3812-c82c-4e4f-8cf3-571265f1629a\") " pod="openshift-monitoring/node-exporter-wrqmt" Nov 04 04:46:18.816919 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:18.816869 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b77f3812-c82c-4e4f-8cf3-571265f1629a-root\") pod \"node-exporter-wrqmt\" (UID: \"b77f3812-c82c-4e4f-8cf3-571265f1629a\") " pod="openshift-monitoring/node-exporter-wrqmt" Nov 04 04:46:18.816919 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:18.816887 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b77f3812-c82c-4e4f-8cf3-571265f1629a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wrqmt\" (UID: \"b77f3812-c82c-4e4f-8cf3-571265f1629a\") " pod="openshift-monitoring/node-exporter-wrqmt" Nov 04 04:46:18.816919 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:18.816914 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b77f3812-c82c-4e4f-8cf3-571265f1629a-node-exporter-textfile\") pod \"node-exporter-wrqmt\" (UID: \"b77f3812-c82c-4e4f-8cf3-571265f1629a\") " pod="openshift-monitoring/node-exporter-wrqmt" Nov 04 04:46:18.817164 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:18.816919 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b77f3812-c82c-4e4f-8cf3-571265f1629a-sys\") pod \"node-exporter-wrqmt\" (UID: \"b77f3812-c82c-4e4f-8cf3-571265f1629a\") " pod="openshift-monitoring/node-exporter-wrqmt" Nov 04 04:46:18.817164 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:18.816936 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b77f3812-c82c-4e4f-8cf3-571265f1629a-metrics-client-ca\") pod \"node-exporter-wrqmt\" (UID: \"b77f3812-c82c-4e4f-8cf3-571265f1629a\") " pod="openshift-monitoring/node-exporter-wrqmt" Nov 04 04:46:18.817164 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:18.816928 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b77f3812-c82c-4e4f-8cf3-571265f1629a-root\") pod \"node-exporter-wrqmt\" (UID: \"b77f3812-c82c-4e4f-8cf3-571265f1629a\") " pod="openshift-monitoring/node-exporter-wrqmt" Nov 04 04:46:18.817164 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:18.816980 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b77f3812-c82c-4e4f-8cf3-571265f1629a-node-exporter-wtmp\") pod \"node-exporter-wrqmt\" (UID: \"b77f3812-c82c-4e4f-8cf3-571265f1629a\") " pod="openshift-monitoring/node-exporter-wrqmt" Nov 04 04:46:18.817164 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:18.816991 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrh47\" (UniqueName: \"kubernetes.io/projected/b77f3812-c82c-4e4f-8cf3-571265f1629a-kube-api-access-wrh47\") pod \"node-exporter-wrqmt\" (UID: \"b77f3812-c82c-4e4f-8cf3-571265f1629a\") " pod="openshift-monitoring/node-exporter-wrqmt" Nov 04 04:46:18.817359 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:18.817213 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b77f3812-c82c-4e4f-8cf3-571265f1629a-node-exporter-textfile\") pod \"node-exporter-wrqmt\" (UID: \"b77f3812-c82c-4e4f-8cf3-571265f1629a\") " pod="openshift-monitoring/node-exporter-wrqmt" Nov 04 04:46:18.817417 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:18.817402 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b77f3812-c82c-4e4f-8cf3-571265f1629a-metrics-client-ca\") pod \"node-exporter-wrqmt\" (UID: \"b77f3812-c82c-4e4f-8cf3-571265f1629a\") " pod="openshift-monitoring/node-exporter-wrqmt" Nov 04 04:46:18.832421 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:18.832402 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 04 04:46:18.832623 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:18.832608 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 04 04:46:18.832700 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:18.832689 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mjbxn\"" Nov 04 04:46:18.832754 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:18.832739 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 04 04:46:18.838250 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:18.838223 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b77f3812-c82c-4e4f-8cf3-571265f1629a-node-exporter-accelerators-collector-config\") pod \"node-exporter-wrqmt\" (UID: \"b77f3812-c82c-4e4f-8cf3-571265f1629a\") " pod="openshift-monitoring/node-exporter-wrqmt" Nov 04 04:46:18.839758 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:18.839741 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b77f3812-c82c-4e4f-8cf3-571265f1629a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wrqmt\" (UID: \"b77f3812-c82c-4e4f-8cf3-571265f1629a\") " pod="openshift-monitoring/node-exporter-wrqmt" Nov 04 04:46:18.839933 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:18.839915 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b77f3812-c82c-4e4f-8cf3-571265f1629a-node-exporter-tls\") pod \"node-exporter-wrqmt\" (UID: \"b77f3812-c82c-4e4f-8cf3-571265f1629a\") " pod="openshift-monitoring/node-exporter-wrqmt" Nov 04 04:46:18.874464 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:18.874439 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrh47\" (UniqueName: \"kubernetes.io/projected/b77f3812-c82c-4e4f-8cf3-571265f1629a-kube-api-access-wrh47\") pod \"node-exporter-wrqmt\" (UID: \"b77f3812-c82c-4e4f-8cf3-571265f1629a\") " pod="openshift-monitoring/node-exporter-wrqmt" Nov 04 04:46:18.994685 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:18.994663 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wrqmt" Nov 04 04:46:19.007340 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:46:19.007310 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb77f3812_c82c_4e4f_8cf3_571265f1629a.slice/crio-fd7694e0ef3da980fd707337040d3e68fb2e874cbb3fc236929c43e6ab42ddea WatchSource:0}: Error finding container fd7694e0ef3da980fd707337040d3e68fb2e874cbb3fc236929c43e6ab42ddea: Status 404 returned error can't find the container with id fd7694e0ef3da980fd707337040d3e68fb2e874cbb3fc236929c43e6ab42ddea Nov 04 04:46:19.517957 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:19.517920 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wrqmt" event={"ID":"b77f3812-c82c-4e4f-8cf3-571265f1629a","Type":"ContainerStarted","Data":"fd7694e0ef3da980fd707337040d3e68fb2e874cbb3fc236929c43e6ab42ddea"} Nov 04 04:46:21.311580 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.311549 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-75dcc4968f-ql7lm"] Nov 04 04:46:21.314779 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.314763 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-75dcc4968f-ql7lm" Nov 04 04:46:21.331409 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.331389 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-75dcc4968f-ql7lm"] Nov 04 04:46:21.377481 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.377459 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 04 04:46:21.377592 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.377463 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 04 04:46:21.377592 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.377492 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-orqae72pteo3\"" Nov 04 04:46:21.377592 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.377463 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 04 04:46:21.377592 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.377463 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 04 04:46:21.377592 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.377515 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 04 04:46:21.377592 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.377585 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-xf5gl\"" Nov 04 04:46:21.433185 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.433161 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/369386da-4587-45d0-b344-0809b01f162b-secret-grpc-tls\") pod \"thanos-querier-75dcc4968f-ql7lm\" (UID: \"369386da-4587-45d0-b344-0809b01f162b\") " pod="openshift-monitoring/thanos-querier-75dcc4968f-ql7lm" Nov 04 04:46:21.433185 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.433185 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdrs8\" (UniqueName: \"kubernetes.io/projected/369386da-4587-45d0-b344-0809b01f162b-kube-api-access-bdrs8\") pod \"thanos-querier-75dcc4968f-ql7lm\" (UID: \"369386da-4587-45d0-b344-0809b01f162b\") " pod="openshift-monitoring/thanos-querier-75dcc4968f-ql7lm" Nov 04 04:46:21.433357 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.433203 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/369386da-4587-45d0-b344-0809b01f162b-metrics-client-ca\") pod \"thanos-querier-75dcc4968f-ql7lm\" (UID: \"369386da-4587-45d0-b344-0809b01f162b\") " pod="openshift-monitoring/thanos-querier-75dcc4968f-ql7lm" Nov 04 04:46:21.433357 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.433229 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/369386da-4587-45d0-b344-0809b01f162b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-75dcc4968f-ql7lm\" (UID: \"369386da-4587-45d0-b344-0809b01f162b\") " pod="openshift-monitoring/thanos-querier-75dcc4968f-ql7lm" Nov 04 04:46:21.433357 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.433279 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/369386da-4587-45d0-b344-0809b01f162b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-75dcc4968f-ql7lm\" (UID: \"369386da-4587-45d0-b344-0809b01f162b\") " pod="openshift-monitoring/thanos-querier-75dcc4968f-ql7lm" Nov 04 04:46:21.433471 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.433363 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/369386da-4587-45d0-b344-0809b01f162b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-75dcc4968f-ql7lm\" (UID: \"369386da-4587-45d0-b344-0809b01f162b\") " pod="openshift-monitoring/thanos-querier-75dcc4968f-ql7lm" Nov 04 04:46:21.433471 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.433385 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/369386da-4587-45d0-b344-0809b01f162b-tmp\") pod \"thanos-querier-75dcc4968f-ql7lm\" (UID: \"369386da-4587-45d0-b344-0809b01f162b\") " pod="openshift-monitoring/thanos-querier-75dcc4968f-ql7lm" Nov 04 04:46:21.433471 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.433453 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/369386da-4587-45d0-b344-0809b01f162b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-75dcc4968f-ql7lm\" (UID: \"369386da-4587-45d0-b344-0809b01f162b\") " pod="openshift-monitoring/thanos-querier-75dcc4968f-ql7lm" Nov 04 04:46:21.433567 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.433472 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/369386da-4587-45d0-b344-0809b01f162b-secret-thanos-querier-tls\") pod \"thanos-querier-75dcc4968f-ql7lm\" (UID: \"369386da-4587-45d0-b344-0809b01f162b\") " pod="openshift-monitoring/thanos-querier-75dcc4968f-ql7lm" Nov 04 04:46:21.525767 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.525742 4395 generic.go:356] "Generic (PLEG): container finished" podID="b77f3812-c82c-4e4f-8cf3-571265f1629a" containerID="c147d6b28fbd170d108345e7eb420233685cabda67f6b7ebc878d2352630a4a8" exitCode=0 Nov 04 04:46:21.525866 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.525830 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wrqmt" event={"ID":"b77f3812-c82c-4e4f-8cf3-571265f1629a","Type":"ContainerDied","Data":"c147d6b28fbd170d108345e7eb420233685cabda67f6b7ebc878d2352630a4a8"} Nov 04 04:46:21.534626 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.534598 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/369386da-4587-45d0-b344-0809b01f162b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-75dcc4968f-ql7lm\" (UID: \"369386da-4587-45d0-b344-0809b01f162b\") " pod="openshift-monitoring/thanos-querier-75dcc4968f-ql7lm" Nov 04 04:46:21.534728 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.534631 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/369386da-4587-45d0-b344-0809b01f162b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-75dcc4968f-ql7lm\" (UID: \"369386da-4587-45d0-b344-0809b01f162b\") " pod="openshift-monitoring/thanos-querier-75dcc4968f-ql7lm" Nov 04 04:46:21.534728 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.534653 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/369386da-4587-45d0-b344-0809b01f162b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-75dcc4968f-ql7lm\" (UID: \"369386da-4587-45d0-b344-0809b01f162b\") " pod="openshift-monitoring/thanos-querier-75dcc4968f-ql7lm" Nov 04 04:46:21.534837 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.534784 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/369386da-4587-45d0-b344-0809b01f162b-tmp\") pod \"thanos-querier-75dcc4968f-ql7lm\" (UID: \"369386da-4587-45d0-b344-0809b01f162b\") " pod="openshift-monitoring/thanos-querier-75dcc4968f-ql7lm" Nov 04 04:46:21.534887 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.534841 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/369386da-4587-45d0-b344-0809b01f162b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-75dcc4968f-ql7lm\" (UID: \"369386da-4587-45d0-b344-0809b01f162b\") " pod="openshift-monitoring/thanos-querier-75dcc4968f-ql7lm" Nov 04 04:46:21.534887 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.534873 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/369386da-4587-45d0-b344-0809b01f162b-secret-thanos-querier-tls\") pod \"thanos-querier-75dcc4968f-ql7lm\" (UID: \"369386da-4587-45d0-b344-0809b01f162b\") " pod="openshift-monitoring/thanos-querier-75dcc4968f-ql7lm" Nov 04 04:46:21.534986 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.534913 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/369386da-4587-45d0-b344-0809b01f162b-secret-grpc-tls\") pod \"thanos-querier-75dcc4968f-ql7lm\" (UID: \"369386da-4587-45d0-b344-0809b01f162b\") " pod="openshift-monitoring/thanos-querier-75dcc4968f-ql7lm" Nov 04 04:46:21.535096 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.535060 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdrs8\" (UniqueName: \"kubernetes.io/projected/369386da-4587-45d0-b344-0809b01f162b-kube-api-access-bdrs8\") pod \"thanos-querier-75dcc4968f-ql7lm\" (UID: \"369386da-4587-45d0-b344-0809b01f162b\") " pod="openshift-monitoring/thanos-querier-75dcc4968f-ql7lm" Nov 04 04:46:21.535096 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.535086 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/369386da-4587-45d0-b344-0809b01f162b-metrics-client-ca\") pod \"thanos-querier-75dcc4968f-ql7lm\" (UID: \"369386da-4587-45d0-b344-0809b01f162b\") " pod="openshift-monitoring/thanos-querier-75dcc4968f-ql7lm" Nov 04 04:46:21.535524 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.535208 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/369386da-4587-45d0-b344-0809b01f162b-tmp\") pod \"thanos-querier-75dcc4968f-ql7lm\" (UID: \"369386da-4587-45d0-b344-0809b01f162b\") " pod="openshift-monitoring/thanos-querier-75dcc4968f-ql7lm" Nov 04 04:46:21.535941 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.535914 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/369386da-4587-45d0-b344-0809b01f162b-metrics-client-ca\") pod \"thanos-querier-75dcc4968f-ql7lm\" (UID: \"369386da-4587-45d0-b344-0809b01f162b\") " pod="openshift-monitoring/thanos-querier-75dcc4968f-ql7lm" Nov 04 04:46:21.537985 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.537955 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/369386da-4587-45d0-b344-0809b01f162b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-75dcc4968f-ql7lm\" (UID: \"369386da-4587-45d0-b344-0809b01f162b\") " pod="openshift-monitoring/thanos-querier-75dcc4968f-ql7lm" Nov 04 04:46:21.538092 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.537995 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/369386da-4587-45d0-b344-0809b01f162b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-75dcc4968f-ql7lm\" (UID: \"369386da-4587-45d0-b344-0809b01f162b\") " pod="openshift-monitoring/thanos-querier-75dcc4968f-ql7lm" Nov 04 04:46:21.538092 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.538056 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/369386da-4587-45d0-b344-0809b01f162b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-75dcc4968f-ql7lm\" (UID: \"369386da-4587-45d0-b344-0809b01f162b\") " pod="openshift-monitoring/thanos-querier-75dcc4968f-ql7lm" Nov 04 04:46:21.538207 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.538110 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/369386da-4587-45d0-b344-0809b01f162b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-75dcc4968f-ql7lm\" (UID: \"369386da-4587-45d0-b344-0809b01f162b\") " pod="openshift-monitoring/thanos-querier-75dcc4968f-ql7lm" Nov 04 04:46:21.538261 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.538219 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/369386da-4587-45d0-b344-0809b01f162b-secret-grpc-tls\") pod \"thanos-querier-75dcc4968f-ql7lm\" (UID: \"369386da-4587-45d0-b344-0809b01f162b\") " pod="openshift-monitoring/thanos-querier-75dcc4968f-ql7lm" Nov 04 04:46:21.538735 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.538717 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/369386da-4587-45d0-b344-0809b01f162b-secret-thanos-querier-tls\") pod \"thanos-querier-75dcc4968f-ql7lm\" (UID: \"369386da-4587-45d0-b344-0809b01f162b\") " pod="openshift-monitoring/thanos-querier-75dcc4968f-ql7lm" Nov 04 04:46:21.591982 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.591964 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdrs8\" (UniqueName: \"kubernetes.io/projected/369386da-4587-45d0-b344-0809b01f162b-kube-api-access-bdrs8\") pod \"thanos-querier-75dcc4968f-ql7lm\" (UID: \"369386da-4587-45d0-b344-0809b01f162b\") " pod="openshift-monitoring/thanos-querier-75dcc4968f-ql7lm" Nov 04 04:46:21.635497 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:21.635474 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-75dcc4968f-ql7lm" Nov 04 04:46:22.101704 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:22.101679 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-75dcc4968f-ql7lm"] Nov 04 04:46:22.104417 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:46:22.104382 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod369386da_4587_45d0_b344_0809b01f162b.slice/crio-83ed388cff47b62f6be777de7300e75811b5da06c7a62efe082f6046ff113703 WatchSource:0}: Error finding container 83ed388cff47b62f6be777de7300e75811b5da06c7a62efe082f6046ff113703: Status 404 returned error can't find the container with id 83ed388cff47b62f6be777de7300e75811b5da06c7a62efe082f6046ff113703 Nov 04 04:46:22.227152 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:22.227123 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84d44d8fcb-hmt2s" Nov 04 04:46:22.227256 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:22.227176 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84d44d8fcb-hmt2s" Nov 04 04:46:22.232043 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:22.232024 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84d44d8fcb-hmt2s" Nov 04 04:46:22.530780 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:22.530712 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wrqmt" event={"ID":"b77f3812-c82c-4e4f-8cf3-571265f1629a","Type":"ContainerStarted","Data":"a205d78941263e4b0027eb55e7f58a137924da86673a3932fdabdf645ab005e6"} Nov 04 04:46:22.530780 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:22.530757 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wrqmt" event={"ID":"b77f3812-c82c-4e4f-8cf3-571265f1629a","Type":"ContainerStarted","Data":"c5016a68480c6886664ad50b90f9e3ee24f9d1898f9f8ad8c37d803c30045b99"} Nov 04 04:46:22.531949 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:22.531925 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75dcc4968f-ql7lm" event={"ID":"369386da-4587-45d0-b344-0809b01f162b","Type":"ContainerStarted","Data":"83ed388cff47b62f6be777de7300e75811b5da06c7a62efe082f6046ff113703"} Nov 04 04:46:22.535818 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:22.535801 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84d44d8fcb-hmt2s" Nov 04 04:46:22.642506 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:22.642465 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wrqmt" podStartSLOduration=3.013777041 podStartE2EDuration="4.642452129s" totalImagesPullingTime="1.56327406s" totalInitContainerRuntime="65.401028ms" isStatefulPod=true podCreationTimestamp="2026-11-04 04:46:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:46:22.641860293 +0000 UTC m=+24.331942740" watchObservedRunningTime="2026-11-04 04:46:22.642452129 +0000 UTC m=+24.332534576" Nov 04 04:46:23.255602 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:23.255572 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-m4bwb"] Nov 04 04:46:23.259766 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:23.259743 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-m4bwb" Nov 04 04:46:23.316225 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:23.316193 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-m4bwb"] Nov 04 04:46:23.348039 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:23.348004 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/343dcbb2-ac23-4978-a763-03964fd95eaf-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-m4bwb\" (UID: \"343dcbb2-ac23-4978-a763-03964fd95eaf\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-m4bwb" Nov 04 04:46:23.348160 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:23.348059 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcxfd\" (UniqueName: \"kubernetes.io/projected/343dcbb2-ac23-4978-a763-03964fd95eaf-kube-api-access-gcxfd\") pod \"monitoring-plugin-77f4f76594-m4bwb\" (UID: \"343dcbb2-ac23-4978-a763-03964fd95eaf\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-m4bwb" Nov 04 04:46:23.348838 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:23.348820 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-f7wjx\"" Nov 04 04:46:23.348838 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:23.348832 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 04 04:46:23.448656 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:23.448635 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/343dcbb2-ac23-4978-a763-03964fd95eaf-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-m4bwb\" (UID: \"343dcbb2-ac23-4978-a763-03964fd95eaf\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-m4bwb" Nov 04 04:46:23.448784 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:23.448676 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcxfd\" (UniqueName: \"kubernetes.io/projected/343dcbb2-ac23-4978-a763-03964fd95eaf-kube-api-access-gcxfd\") pod \"monitoring-plugin-77f4f76594-m4bwb\" (UID: \"343dcbb2-ac23-4978-a763-03964fd95eaf\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-m4bwb" Nov 04 04:46:23.451362 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:23.451343 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/343dcbb2-ac23-4978-a763-03964fd95eaf-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-m4bwb\" (UID: \"343dcbb2-ac23-4978-a763-03964fd95eaf\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-m4bwb" Nov 04 04:46:23.505992 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:23.505944 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcxfd\" (UniqueName: \"kubernetes.io/projected/343dcbb2-ac23-4978-a763-03964fd95eaf-kube-api-access-gcxfd\") pod \"monitoring-plugin-77f4f76594-m4bwb\" (UID: \"343dcbb2-ac23-4978-a763-03964fd95eaf\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-m4bwb" Nov 04 04:46:23.571008 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:23.570985 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-m4bwb" Nov 04 04:46:23.934994 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:23.934964 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-m4bwb"] Nov 04 04:46:23.937981 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:46:23.937946 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod343dcbb2_ac23_4978_a763_03964fd95eaf.slice/crio-d2d1d5559aad640cd60cc7ad74ab87859ac1bf7ad55101a60e1e990fba8c244e WatchSource:0}: Error finding container d2d1d5559aad640cd60cc7ad74ab87859ac1bf7ad55101a60e1e990fba8c244e: Status 404 returned error can't find the container with id d2d1d5559aad640cd60cc7ad74ab87859ac1bf7ad55101a60e1e990fba8c244e Nov 04 04:46:24.539327 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:24.539281 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-m4bwb" event={"ID":"343dcbb2-ac23-4978-a763-03964fd95eaf","Type":"ContainerStarted","Data":"d2d1d5559aad640cd60cc7ad74ab87859ac1bf7ad55101a60e1e990fba8c244e"} Nov 04 04:46:25.506796 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:25.506766 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hs4hb" Nov 04 04:46:26.129873 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:26.129833 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mfn6w" Nov 04 04:46:26.362502 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:26.361922 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ct2mt" Nov 04 04:46:26.547403 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:26.547370 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-m4bwb" event={"ID":"343dcbb2-ac23-4978-a763-03964fd95eaf","Type":"ContainerStarted","Data":"df30c906d2236262de195e4ac2dd3b8668a9592e3a1355005c4e51eb523bce90"} Nov 04 04:46:26.547865 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:26.547543 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-m4bwb" Nov 04 04:46:26.552091 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:26.552062 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-m4bwb" Nov 04 04:46:26.658038 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:26.657996 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-m4bwb" podStartSLOduration=1.244822785 podStartE2EDuration="3.657983549s" totalImagesPullingTime="2.413160764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:46:26.657585247 +0000 UTC m=+28.347667703" watchObservedRunningTime="2026-11-04 04:46:26.657983549 +0000 UTC m=+28.348065990" Nov 04 04:46:27.554401 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:27.554364 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75dcc4968f-ql7lm" event={"ID":"369386da-4587-45d0-b344-0809b01f162b","Type":"ContainerStarted","Data":"a0e48a3410cce127952ab5df1b98f19620df0b1598658a61fdba8b39230e20bb"} Nov 04 04:46:27.554787 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:27.554448 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75dcc4968f-ql7lm" event={"ID":"369386da-4587-45d0-b344-0809b01f162b","Type":"ContainerStarted","Data":"1a618a709590a1f668418f280a9dab07a64aa58cc59ec92fbc1354eb33fa50ca"} Nov 04 04:46:27.554787 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:27.554495 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75dcc4968f-ql7lm" event={"ID":"369386da-4587-45d0-b344-0809b01f162b","Type":"ContainerStarted","Data":"b126de40a602d71467153bd32b084348218ca5caf8ee918db13cc2a830a94101"} Nov 04 04:46:27.935138 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:27.935045 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 04:46:27.939022 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:27.939001 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:27.982037 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:27.982007 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:27.982140 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:27.982043 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:27.982140 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:27.982062 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:27.982140 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:27.982114 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-config-out\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:27.982266 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:27.982243 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:27.982266 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:27.982263 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:27.982365 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:27.982278 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:27.982365 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:27.982293 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:27.982365 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:27.982360 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-web-config\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:27.982451 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:27.982389 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:27.982451 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:27.982415 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:27.982451 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:27.982439 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:27.982547 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:27.982538 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:27.982589 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:27.982574 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:27.982619 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:27.982598 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-952s7\" (UniqueName: \"kubernetes.io/projected/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-kube-api-access-952s7\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:27.982670 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:27.982618 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-config\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:27.982670 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:27.982645 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:27.982764 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:27.982723 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:27.982818 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:27.982764 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:27.982875 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:27.982814 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:27.988320 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:27.988247 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 04 04:46:27.988320 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:27.988258 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 04 04:46:27.988320 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:27.988280 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 04 04:46:27.988471 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:27.988408 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 04 04:46:27.988509 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:27.988498 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 04 04:46:27.988553 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:27.988538 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 04 04:46:28.034479 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.034455 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 04 04:46:28.034593 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.034523 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 04 04:46:28.034593 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.034536 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ca9nc2q72skrd\"" Nov 04 04:46:28.034593 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.034543 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2cm5h\"" Nov 04 04:46:28.034593 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.034557 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 04 04:46:28.034761 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.034652 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 04 04:46:28.083138 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.083109 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.083261 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.083145 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.083261 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.083168 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-952s7\" (UniqueName: \"kubernetes.io/projected/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-kube-api-access-952s7\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.083390 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.083290 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-config\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.083390 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.083350 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.083390 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.083383 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.083548 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.083408 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.083548 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.083440 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.083548 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.083478 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.083548 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.083502 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.083548 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.083526 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.084003 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.083574 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-config-out\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.084003 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.083606 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.084003 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.083658 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.084003 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.083670 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.084003 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.083720 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.084003 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.083745 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.084003 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.083826 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-web-config\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.084003 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.083875 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.084003 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.083920 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.084003 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.083944 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.084512 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.084118 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.084512 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.084329 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.084677 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.084655 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.085315 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.085271 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.085599 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.085574 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.086718 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.086695 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-web-config\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.086829 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.086813 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-config-out\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.086961 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.086938 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.087757 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.087736 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.087945 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.087926 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.087994 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.087974 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-config\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.088056 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.088042 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.088356 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.088337 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.127766 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.127744 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 04 04:46:28.127994 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.127980 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 04 04:46:28.128083 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.128059 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 04 04:46:28.128212 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.128117 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 04 04:46:28.128760 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.128738 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 04:46:28.137017 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.137000 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.137243 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.137224 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.137283 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.137266 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.137442 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.137428 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.174392 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.174371 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 04 04:46:28.176507 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.176482 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-952s7\" (UniqueName: \"kubernetes.io/projected/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-kube-api-access-952s7\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.185371 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.185318 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.268790 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.268771 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:28.691734 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:28.691713 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 04:46:28.694329 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:46:28.694249 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ef6a31_f947_4b17_90ba_5d16957f9ea1.slice/crio-28470ee02cff26b5794370760347764b1cc84a135e9df9eb6906727cdb431219 WatchSource:0}: Error finding container 28470ee02cff26b5794370760347764b1cc84a135e9df9eb6906727cdb431219: Status 404 returned error can't find the container with id 28470ee02cff26b5794370760347764b1cc84a135e9df9eb6906727cdb431219 Nov 04 04:46:29.563617 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:29.563584 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75dcc4968f-ql7lm" event={"ID":"369386da-4587-45d0-b344-0809b01f162b","Type":"ContainerStarted","Data":"25cb5d01d7d039bd9ae2b21c325dce0948c6f814e60fe2d70d52df367a3924d0"} Nov 04 04:46:29.563793 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:29.563693 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75dcc4968f-ql7lm" event={"ID":"369386da-4587-45d0-b344-0809b01f162b","Type":"ContainerStarted","Data":"a62cc0911e748b25b712a323a83dddb293118f80662b47ed5fad4f838fbf69d0"} Nov 04 04:46:29.563793 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:29.563733 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-75dcc4968f-ql7lm" Nov 04 04:46:29.563793 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:29.563770 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75dcc4968f-ql7lm" event={"ID":"369386da-4587-45d0-b344-0809b01f162b","Type":"ContainerStarted","Data":"7b10172884fc7ac12203f462c0556e57249484ad35514f957fc78bfeadaf7308"} Nov 04 04:46:29.564882 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:29.564861 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1ef6a31-f947-4b17-90ba-5d16957f9ea1","Type":"ContainerStarted","Data":"28470ee02cff26b5794370760347764b1cc84a135e9df9eb6906727cdb431219"} Nov 04 04:46:29.681152 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:29.681091 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-75dcc4968f-ql7lm" podStartSLOduration=2.000276709 podStartE2EDuration="8.681072475s" totalImagesPullingTime="6.680795766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:46:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:46:29.679639072 +0000 UTC m=+31.369721520" watchObservedRunningTime="2026-11-04 04:46:29.681072475 +0000 UTC m=+31.371154922" Nov 04 04:46:31.573482 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:31.573454 4395 generic.go:356] "Generic (PLEG): container finished" podID="a1ef6a31-f947-4b17-90ba-5d16957f9ea1" containerID="528ff97e2117bb6babeef289a032fdad24c21dc56c710085a27f08d2a83a2dfc" exitCode=0 Nov 04 04:46:31.573818 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:31.573499 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1ef6a31-f947-4b17-90ba-5d16957f9ea1","Type":"ContainerDied","Data":"528ff97e2117bb6babeef289a032fdad24c21dc56c710085a27f08d2a83a2dfc"} Nov 04 04:46:33.456234 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:46:33.456194 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 04:46:35.575196 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:35.575171 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-75dcc4968f-ql7lm" Nov 04 04:46:36.766576 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:36.766549 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84d44d8fcb-hmt2s"] Nov 04 04:46:37.029240 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:37.029217 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-b478c5787-vgmt8"] Nov 04 04:46:37.035837 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:37.035812 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b478c5787-vgmt8" Nov 04 04:46:37.137682 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:37.137628 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 04 04:46:37.138392 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:37.138375 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b478c5787-vgmt8"] Nov 04 04:46:37.154232 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:37.154206 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f16979cc-64df-4df4-9498-70d87f82d236-oauth-serving-cert\") pod \"console-b478c5787-vgmt8\" (UID: \"f16979cc-64df-4df4-9498-70d87f82d236\") " pod="openshift-console/console-b478c5787-vgmt8" Nov 04 04:46:37.154353 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:37.154239 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f16979cc-64df-4df4-9498-70d87f82d236-trusted-ca-bundle\") pod \"console-b478c5787-vgmt8\" (UID: \"f16979cc-64df-4df4-9498-70d87f82d236\") " pod="openshift-console/console-b478c5787-vgmt8" Nov 04 04:46:37.154353 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:37.154262 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f16979cc-64df-4df4-9498-70d87f82d236-console-config\") pod \"console-b478c5787-vgmt8\" (UID: \"f16979cc-64df-4df4-9498-70d87f82d236\") " pod="openshift-console/console-b478c5787-vgmt8" Nov 04 04:46:37.154353 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:37.154346 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f16979cc-64df-4df4-9498-70d87f82d236-console-oauth-config\") pod \"console-b478c5787-vgmt8\" (UID: \"f16979cc-64df-4df4-9498-70d87f82d236\") " pod="openshift-console/console-b478c5787-vgmt8" Nov 04 04:46:37.154482 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:37.154397 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f16979cc-64df-4df4-9498-70d87f82d236-console-serving-cert\") pod \"console-b478c5787-vgmt8\" (UID: \"f16979cc-64df-4df4-9498-70d87f82d236\") " pod="openshift-console/console-b478c5787-vgmt8" Nov 04 04:46:37.154482 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:37.154433 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f16979cc-64df-4df4-9498-70d87f82d236-service-ca\") pod \"console-b478c5787-vgmt8\" (UID: \"f16979cc-64df-4df4-9498-70d87f82d236\") " pod="openshift-console/console-b478c5787-vgmt8" Nov 04 04:46:37.154482 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:37.154462 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f16979cc-64df-4df4-9498-70d87f82d236-tmp\") pod \"console-b478c5787-vgmt8\" (UID: \"f16979cc-64df-4df4-9498-70d87f82d236\") " pod="openshift-console/console-b478c5787-vgmt8" Nov 04 04:46:37.154579 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:37.154504 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6k8d\" (UniqueName: \"kubernetes.io/projected/f16979cc-64df-4df4-9498-70d87f82d236-kube-api-access-t6k8d\") pod \"console-b478c5787-vgmt8\" (UID: \"f16979cc-64df-4df4-9498-70d87f82d236\") " pod="openshift-console/console-b478c5787-vgmt8" Nov 04 04:46:37.255648 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:37.255623 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f16979cc-64df-4df4-9498-70d87f82d236-console-config\") pod \"console-b478c5787-vgmt8\" (UID: \"f16979cc-64df-4df4-9498-70d87f82d236\") " pod="openshift-console/console-b478c5787-vgmt8" Nov 04 04:46:37.255768 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:37.255656 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f16979cc-64df-4df4-9498-70d87f82d236-console-oauth-config\") pod \"console-b478c5787-vgmt8\" (UID: \"f16979cc-64df-4df4-9498-70d87f82d236\") " pod="openshift-console/console-b478c5787-vgmt8" Nov 04 04:46:37.255768 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:37.255675 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f16979cc-64df-4df4-9498-70d87f82d236-console-serving-cert\") pod \"console-b478c5787-vgmt8\" (UID: \"f16979cc-64df-4df4-9498-70d87f82d236\") " pod="openshift-console/console-b478c5787-vgmt8" Nov 04 04:46:37.255768 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:37.255696 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f16979cc-64df-4df4-9498-70d87f82d236-service-ca\") pod \"console-b478c5787-vgmt8\" (UID: \"f16979cc-64df-4df4-9498-70d87f82d236\") " pod="openshift-console/console-b478c5787-vgmt8" Nov 04 04:46:37.255768 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:37.255729 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f16979cc-64df-4df4-9498-70d87f82d236-tmp\") pod \"console-b478c5787-vgmt8\" (UID: \"f16979cc-64df-4df4-9498-70d87f82d236\") " pod="openshift-console/console-b478c5787-vgmt8" Nov 04 04:46:37.256244 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:37.255783 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6k8d\" (UniqueName: \"kubernetes.io/projected/f16979cc-64df-4df4-9498-70d87f82d236-kube-api-access-t6k8d\") pod \"console-b478c5787-vgmt8\" (UID: \"f16979cc-64df-4df4-9498-70d87f82d236\") " pod="openshift-console/console-b478c5787-vgmt8" Nov 04 04:46:37.256244 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:37.255834 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f16979cc-64df-4df4-9498-70d87f82d236-oauth-serving-cert\") pod \"console-b478c5787-vgmt8\" (UID: \"f16979cc-64df-4df4-9498-70d87f82d236\") " pod="openshift-console/console-b478c5787-vgmt8" Nov 04 04:46:37.256244 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:37.255872 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f16979cc-64df-4df4-9498-70d87f82d236-trusted-ca-bundle\") pod \"console-b478c5787-vgmt8\" (UID: \"f16979cc-64df-4df4-9498-70d87f82d236\") " pod="openshift-console/console-b478c5787-vgmt8" Nov 04 04:46:37.256244 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:37.256185 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f16979cc-64df-4df4-9498-70d87f82d236-tmp\") pod \"console-b478c5787-vgmt8\" (UID: \"f16979cc-64df-4df4-9498-70d87f82d236\") " pod="openshift-console/console-b478c5787-vgmt8" Nov 04 04:46:37.256515 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:37.256487 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f16979cc-64df-4df4-9498-70d87f82d236-console-config\") pod \"console-b478c5787-vgmt8\" (UID: \"f16979cc-64df-4df4-9498-70d87f82d236\") " pod="openshift-console/console-b478c5787-vgmt8" Nov 04 04:46:37.256653 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:37.256633 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f16979cc-64df-4df4-9498-70d87f82d236-service-ca\") pod \"console-b478c5787-vgmt8\" (UID: \"f16979cc-64df-4df4-9498-70d87f82d236\") " pod="openshift-console/console-b478c5787-vgmt8" Nov 04 04:46:37.256779 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:37.256761 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f16979cc-64df-4df4-9498-70d87f82d236-oauth-serving-cert\") pod \"console-b478c5787-vgmt8\" (UID: \"f16979cc-64df-4df4-9498-70d87f82d236\") " pod="openshift-console/console-b478c5787-vgmt8" Nov 04 04:46:37.256871 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:37.256849 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f16979cc-64df-4df4-9498-70d87f82d236-trusted-ca-bundle\") pod \"console-b478c5787-vgmt8\" (UID: \"f16979cc-64df-4df4-9498-70d87f82d236\") " pod="openshift-console/console-b478c5787-vgmt8" Nov 04 04:46:37.258273 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:37.258250 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f16979cc-64df-4df4-9498-70d87f82d236-console-serving-cert\") pod \"console-b478c5787-vgmt8\" (UID: \"f16979cc-64df-4df4-9498-70d87f82d236\") " pod="openshift-console/console-b478c5787-vgmt8" Nov 04 04:46:37.258353 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:37.258324 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f16979cc-64df-4df4-9498-70d87f82d236-console-oauth-config\") pod \"console-b478c5787-vgmt8\" (UID: \"f16979cc-64df-4df4-9498-70d87f82d236\") " pod="openshift-console/console-b478c5787-vgmt8" Nov 04 04:46:37.312955 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:37.312933 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6k8d\" (UniqueName: \"kubernetes.io/projected/f16979cc-64df-4df4-9498-70d87f82d236-kube-api-access-t6k8d\") pod \"console-b478c5787-vgmt8\" (UID: \"f16979cc-64df-4df4-9498-70d87f82d236\") " pod="openshift-console/console-b478c5787-vgmt8" Nov 04 04:46:37.345716 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:37.345695 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b478c5787-vgmt8" Nov 04 04:46:37.598325 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:37.598263 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1ef6a31-f947-4b17-90ba-5d16957f9ea1","Type":"ContainerStarted","Data":"71549b35337120be7ee09e530c88cec5725ddd1bbbb36cbb7f861be325217157"} Nov 04 04:46:37.598482 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:37.598372 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1ef6a31-f947-4b17-90ba-5d16957f9ea1","Type":"ContainerStarted","Data":"cb7838e8ec53d290b36e440deffbb2d765c09104248c224e7594d6b43cdd5bea"} Nov 04 04:46:37.598482 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:37.598418 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1ef6a31-f947-4b17-90ba-5d16957f9ea1","Type":"ContainerStarted","Data":"fef92cb15f635b1ee9fe946cf12740dff3fd7b628be9b943d196e5909b057cb9"} Nov 04 04:46:37.598482 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:37.598461 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1ef6a31-f947-4b17-90ba-5d16957f9ea1","Type":"ContainerStarted","Data":"e7a19405d75261ef50d459ad8359da6da340708cc670b049efd2ac7b7a328525"} Nov 04 04:46:37.598582 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:37.598500 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1ef6a31-f947-4b17-90ba-5d16957f9ea1","Type":"ContainerStarted","Data":"50de7fa737b496c79942243e7cfdc8e0c58761e0b493506b8944e58d7009af81"} Nov 04 04:46:37.598582 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:37.598544 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1ef6a31-f947-4b17-90ba-5d16957f9ea1","Type":"ContainerStarted","Data":"d4a4beba708596d4f6159dd0193793ea0dcc87e860cc968acfcc06a06540b538"} Nov 04 04:46:37.707846 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:37.707639 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b478c5787-vgmt8"] Nov 04 04:46:37.710146 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:46:37.710123 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf16979cc_64df_4df4_9498_70d87f82d236.slice/crio-474aea2e1e66f95658f8d6a02d1a71c518265d4805e31216219215781ea48b5b WatchSource:0}: Error finding container 474aea2e1e66f95658f8d6a02d1a71c518265d4805e31216219215781ea48b5b: Status 404 returned error can't find the container with id 474aea2e1e66f95658f8d6a02d1a71c518265d4805e31216219215781ea48b5b Nov 04 04:46:37.723142 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:37.723107 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.484704458 podStartE2EDuration="10.723094065s" totalImagesPullingTime="7.215837178s" totalInitContainerRuntime="22.552429ms" isStatefulPod=false podCreationTimestamp="2026-11-04 04:46:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:46:37.722212388 +0000 UTC m=+39.412294876" watchObservedRunningTime="2026-11-04 04:46:37.723094065 +0000 UTC m=+39.413176522" Nov 04 04:46:38.269832 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:38.269801 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:46:38.606181 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:38.606144 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b478c5787-vgmt8" event={"ID":"f16979cc-64df-4df4-9498-70d87f82d236","Type":"ContainerStarted","Data":"d69cab3ff728d037e078b4611452902009617a4230d8b8c47cf25d0142ce7209"} Nov 04 04:46:38.606386 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:38.606220 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b478c5787-vgmt8" event={"ID":"f16979cc-64df-4df4-9498-70d87f82d236","Type":"ContainerStarted","Data":"474aea2e1e66f95658f8d6a02d1a71c518265d4805e31216219215781ea48b5b"} Nov 04 04:46:38.720311 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:38.720256 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-b478c5787-vgmt8" podStartSLOduration=2.72024329 podStartE2EDuration="2.72024329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:46:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:46:38.719624702 +0000 UTC m=+40.409707150" watchObservedRunningTime="2026-11-04 04:46:38.72024329 +0000 UTC m=+40.410325741" Nov 04 04:46:47.348971 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:47.348938 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b478c5787-vgmt8" Nov 04 04:46:47.349485 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:47.349079 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b478c5787-vgmt8" Nov 04 04:46:47.353605 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:47.353584 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b478c5787-vgmt8" Nov 04 04:46:47.641703 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:46:47.641635 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b478c5787-vgmt8" Nov 04 04:47:01.788524 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:01.788468 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-84d44d8fcb-hmt2s" podUID="abe56d25-e504-4167-8a70-95cc1918b64f" containerName="console" containerID="cri-o://bcdb44ca274c27c312bfb46acf65c7c55dcaa8202ef619983a8714cc4e5d6f2b" gracePeriod=15 Nov 04 04:47:02.030599 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:02.030578 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84d44d8fcb-hmt2s" Nov 04 04:47:02.133228 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:02.133203 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abe56d25-e504-4167-8a70-95cc1918b64f-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/abe56d25-e504-4167-8a70-95cc1918b64f-console-serving-cert\") pod \"abe56d25-e504-4167-8a70-95cc1918b64f\" (UID: \"abe56d25-e504-4167-8a70-95cc1918b64f\") " Nov 04 04:47:02.133344 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:02.133240 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abe56d25-e504-4167-8a70-95cc1918b64f-tmp\" (UniqueName: \"kubernetes.io/empty-dir/abe56d25-e504-4167-8a70-95cc1918b64f-tmp\") pod \"abe56d25-e504-4167-8a70-95cc1918b64f\" (UID: \"abe56d25-e504-4167-8a70-95cc1918b64f\") " Nov 04 04:47:02.133344 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:02.133276 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abe56d25-e504-4167-8a70-95cc1918b64f-service-ca\" (UniqueName: \"kubernetes.io/configmap/abe56d25-e504-4167-8a70-95cc1918b64f-service-ca\") pod \"abe56d25-e504-4167-8a70-95cc1918b64f\" (UID: \"abe56d25-e504-4167-8a70-95cc1918b64f\") " Nov 04 04:47:02.133344 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:02.133319 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abe56d25-e504-4167-8a70-95cc1918b64f-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/abe56d25-e504-4167-8a70-95cc1918b64f-oauth-serving-cert\") pod \"abe56d25-e504-4167-8a70-95cc1918b64f\" (UID: \"abe56d25-e504-4167-8a70-95cc1918b64f\") " Nov 04 04:47:02.133344 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:02.133341 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abe56d25-e504-4167-8a70-95cc1918b64f-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/abe56d25-e504-4167-8a70-95cc1918b64f-console-oauth-config\") pod \"abe56d25-e504-4167-8a70-95cc1918b64f\" (UID: \"abe56d25-e504-4167-8a70-95cc1918b64f\") " Nov 04 04:47:02.133543 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:02.133360 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abe56d25-e504-4167-8a70-95cc1918b64f-kube-api-access-n675z\" (UniqueName: \"kubernetes.io/projected/abe56d25-e504-4167-8a70-95cc1918b64f-kube-api-access-n675z\") pod \"abe56d25-e504-4167-8a70-95cc1918b64f\" (UID: \"abe56d25-e504-4167-8a70-95cc1918b64f\") " Nov 04 04:47:02.133543 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:02.133407 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abe56d25-e504-4167-8a70-95cc1918b64f-console-config\" (UniqueName: \"kubernetes.io/configmap/abe56d25-e504-4167-8a70-95cc1918b64f-console-config\") pod \"abe56d25-e504-4167-8a70-95cc1918b64f\" (UID: \"abe56d25-e504-4167-8a70-95cc1918b64f\") " Nov 04 04:47:02.133645 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:02.133539 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe56d25-e504-4167-8a70-95cc1918b64f-tmp" pod "abe56d25-e504-4167-8a70-95cc1918b64f" (UID: "abe56d25-e504-4167-8a70-95cc1918b64f"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:47:02.133789 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:02.133765 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abe56d25-e504-4167-8a70-95cc1918b64f-service-ca" pod "abe56d25-e504-4167-8a70-95cc1918b64f" (UID: "abe56d25-e504-4167-8a70-95cc1918b64f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:47:02.133846 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:02.133776 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abe56d25-e504-4167-8a70-95cc1918b64f-oauth-serving-cert" pod "abe56d25-e504-4167-8a70-95cc1918b64f" (UID: "abe56d25-e504-4167-8a70-95cc1918b64f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:47:02.133846 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:02.133813 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abe56d25-e504-4167-8a70-95cc1918b64f-console-config" pod "abe56d25-e504-4167-8a70-95cc1918b64f" (UID: "abe56d25-e504-4167-8a70-95cc1918b64f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:47:02.135652 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:02.135628 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abe56d25-e504-4167-8a70-95cc1918b64f-kube-api-access-n675z" pod "abe56d25-e504-4167-8a70-95cc1918b64f" (UID: "abe56d25-e504-4167-8a70-95cc1918b64f"). InnerVolumeSpecName "kube-api-access-n675z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:47:02.135771 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:02.135753 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abe56d25-e504-4167-8a70-95cc1918b64f-console-oauth-config" pod "abe56d25-e504-4167-8a70-95cc1918b64f" (UID: "abe56d25-e504-4167-8a70-95cc1918b64f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:47:02.135819 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:02.135774 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abe56d25-e504-4167-8a70-95cc1918b64f-console-serving-cert" pod "abe56d25-e504-4167-8a70-95cc1918b64f" (UID: "abe56d25-e504-4167-8a70-95cc1918b64f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:47:02.234363 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:02.234325 4395 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/abe56d25-e504-4167-8a70-95cc1918b64f-service-ca\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:47:02.234363 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:02.234363 4395 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/abe56d25-e504-4167-8a70-95cc1918b64f-oauth-serving-cert\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:47:02.234485 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:02.234373 4395 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/abe56d25-e504-4167-8a70-95cc1918b64f-console-oauth-config\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:47:02.234485 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:02.234383 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n675z\" (UniqueName: \"kubernetes.io/projected/abe56d25-e504-4167-8a70-95cc1918b64f-kube-api-access-n675z\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:47:02.234485 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:02.234392 4395 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/abe56d25-e504-4167-8a70-95cc1918b64f-console-config\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:47:02.234485 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:02.234402 4395 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/abe56d25-e504-4167-8a70-95cc1918b64f-console-serving-cert\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:47:02.234485 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:02.234411 4395 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/abe56d25-e504-4167-8a70-95cc1918b64f-tmp\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:47:02.690277 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:02.690214 4395 generic.go:356] "Generic (PLEG): container finished" podID="abe56d25-e504-4167-8a70-95cc1918b64f" containerID="bcdb44ca274c27c312bfb46acf65c7c55dcaa8202ef619983a8714cc4e5d6f2b" exitCode=0 Nov 04 04:47:02.690277 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:02.690263 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84d44d8fcb-hmt2s" Nov 04 04:47:02.690458 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:02.690291 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84d44d8fcb-hmt2s" event={"ID":"abe56d25-e504-4167-8a70-95cc1918b64f","Type":"ContainerDied","Data":"bcdb44ca274c27c312bfb46acf65c7c55dcaa8202ef619983a8714cc4e5d6f2b"} Nov 04 04:47:02.690458 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:02.690384 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84d44d8fcb-hmt2s" event={"ID":"abe56d25-e504-4167-8a70-95cc1918b64f","Type":"ContainerDied","Data":"6c2b0131f219b4a10ec9d6d5fab8d41821198333a6c9a3216915cd7a563ed05d"} Nov 04 04:47:02.690458 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:02.690415 4395 scope.go:122] "RemoveContainer" containerID="bcdb44ca274c27c312bfb46acf65c7c55dcaa8202ef619983a8714cc4e5d6f2b" Nov 04 04:47:02.698762 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:02.698747 4395 scope.go:122] "RemoveContainer" containerID="bcdb44ca274c27c312bfb46acf65c7c55dcaa8202ef619983a8714cc4e5d6f2b" Nov 04 04:47:02.699032 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:47:02.699010 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcdb44ca274c27c312bfb46acf65c7c55dcaa8202ef619983a8714cc4e5d6f2b\": container with ID starting with bcdb44ca274c27c312bfb46acf65c7c55dcaa8202ef619983a8714cc4e5d6f2b not found: ID does not exist" containerID="bcdb44ca274c27c312bfb46acf65c7c55dcaa8202ef619983a8714cc4e5d6f2b" Nov 04 04:47:02.699077 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:02.699041 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcdb44ca274c27c312bfb46acf65c7c55dcaa8202ef619983a8714cc4e5d6f2b"} err="failed to get container status \"bcdb44ca274c27c312bfb46acf65c7c55dcaa8202ef619983a8714cc4e5d6f2b\": rpc error: code = NotFound desc = could not find container \"bcdb44ca274c27c312bfb46acf65c7c55dcaa8202ef619983a8714cc4e5d6f2b\": container with ID starting with bcdb44ca274c27c312bfb46acf65c7c55dcaa8202ef619983a8714cc4e5d6f2b not found: ID does not exist" Nov 04 04:47:02.843104 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:02.843074 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84d44d8fcb-hmt2s"] Nov 04 04:47:02.889654 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:02.889628 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84d44d8fcb-hmt2s"] Nov 04 04:47:03.456032 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:47:03.456002 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 04:47:04.414207 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:04.414173 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abe56d25-e504-4167-8a70-95cc1918b64f" path="/var/lib/kubelet/pods/abe56d25-e504-4167-8a70-95cc1918b64f/volumes" Nov 04 04:47:24.184989 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:24.184952 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:24.204234 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:24.204212 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:24.777444 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:24.777419 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:33.456841 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:47:33.456802 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 04:47:41.570143 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.570111 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 04:47:41.570562 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.570439 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a1ef6a31-f947-4b17-90ba-5d16957f9ea1" containerName="prometheus" containerID="cri-o://d4a4beba708596d4f6159dd0193793ea0dcc87e860cc968acfcc06a06540b538" gracePeriod=600 Nov 04 04:47:41.570562 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.570490 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a1ef6a31-f947-4b17-90ba-5d16957f9ea1" containerName="kube-rbac-proxy" containerID="cri-o://cb7838e8ec53d290b36e440deffbb2d765c09104248c224e7594d6b43cdd5bea" gracePeriod=600 Nov 04 04:47:41.570680 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.570564 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a1ef6a31-f947-4b17-90ba-5d16957f9ea1" containerName="kube-rbac-proxy-web" containerID="cri-o://fef92cb15f635b1ee9fe946cf12740dff3fd7b628be9b943d196e5909b057cb9" gracePeriod=600 Nov 04 04:47:41.570680 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.570597 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a1ef6a31-f947-4b17-90ba-5d16957f9ea1" containerName="config-reloader" containerID="cri-o://50de7fa737b496c79942243e7cfdc8e0c58761e0b493506b8944e58d7009af81" gracePeriod=600 Nov 04 04:47:41.570680 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.570549 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a1ef6a31-f947-4b17-90ba-5d16957f9ea1" containerName="thanos-sidecar" containerID="cri-o://e7a19405d75261ef50d459ad8359da6da340708cc670b049efd2ac7b7a328525" gracePeriod=600 Nov 04 04:47:41.570680 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.570516 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a1ef6a31-f947-4b17-90ba-5d16957f9ea1" containerName="kube-rbac-proxy-thanos" containerID="cri-o://71549b35337120be7ee09e530c88cec5725ddd1bbbb36cbb7f861be325217157" gracePeriod=600 Nov 04 04:47:41.818944 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.818919 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:41.829503 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.829453 4395 generic.go:356] "Generic (PLEG): container finished" podID="a1ef6a31-f947-4b17-90ba-5d16957f9ea1" containerID="71549b35337120be7ee09e530c88cec5725ddd1bbbb36cbb7f861be325217157" exitCode=0 Nov 04 04:47:41.829503 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.829473 4395 generic.go:356] "Generic (PLEG): container finished" podID="a1ef6a31-f947-4b17-90ba-5d16957f9ea1" containerID="cb7838e8ec53d290b36e440deffbb2d765c09104248c224e7594d6b43cdd5bea" exitCode=0 Nov 04 04:47:41.829503 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.829481 4395 generic.go:356] "Generic (PLEG): container finished" podID="a1ef6a31-f947-4b17-90ba-5d16957f9ea1" containerID="fef92cb15f635b1ee9fe946cf12740dff3fd7b628be9b943d196e5909b057cb9" exitCode=0 Nov 04 04:47:41.829503 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.829489 4395 generic.go:356] "Generic (PLEG): container finished" podID="a1ef6a31-f947-4b17-90ba-5d16957f9ea1" containerID="e7a19405d75261ef50d459ad8359da6da340708cc670b049efd2ac7b7a328525" exitCode=0 Nov 04 04:47:41.829503 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.829500 4395 generic.go:356] "Generic (PLEG): container finished" podID="a1ef6a31-f947-4b17-90ba-5d16957f9ea1" containerID="50de7fa737b496c79942243e7cfdc8e0c58761e0b493506b8944e58d7009af81" exitCode=0 Nov 04 04:47:41.829823 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.829508 4395 generic.go:356] "Generic (PLEG): container finished" podID="a1ef6a31-f947-4b17-90ba-5d16957f9ea1" containerID="d4a4beba708596d4f6159dd0193793ea0dcc87e860cc968acfcc06a06540b538" exitCode=0 Nov 04 04:47:41.829823 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.829519 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:41.829823 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.829536 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1ef6a31-f947-4b17-90ba-5d16957f9ea1","Type":"ContainerDied","Data":"71549b35337120be7ee09e530c88cec5725ddd1bbbb36cbb7f861be325217157"} Nov 04 04:47:41.829823 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.829628 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1ef6a31-f947-4b17-90ba-5d16957f9ea1","Type":"ContainerDied","Data":"cb7838e8ec53d290b36e440deffbb2d765c09104248c224e7594d6b43cdd5bea"} Nov 04 04:47:41.829823 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.829678 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1ef6a31-f947-4b17-90ba-5d16957f9ea1","Type":"ContainerDied","Data":"fef92cb15f635b1ee9fe946cf12740dff3fd7b628be9b943d196e5909b057cb9"} Nov 04 04:47:41.829823 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.829691 4395 scope.go:122] "RemoveContainer" containerID="71549b35337120be7ee09e530c88cec5725ddd1bbbb36cbb7f861be325217157" Nov 04 04:47:41.829823 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.829726 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1ef6a31-f947-4b17-90ba-5d16957f9ea1","Type":"ContainerDied","Data":"e7a19405d75261ef50d459ad8359da6da340708cc670b049efd2ac7b7a328525"} Nov 04 04:47:41.829823 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.829776 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1ef6a31-f947-4b17-90ba-5d16957f9ea1","Type":"ContainerDied","Data":"50de7fa737b496c79942243e7cfdc8e0c58761e0b493506b8944e58d7009af81"} Nov 04 04:47:41.829823 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.829822 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1ef6a31-f947-4b17-90ba-5d16957f9ea1","Type":"ContainerDied","Data":"d4a4beba708596d4f6159dd0193793ea0dcc87e860cc968acfcc06a06540b538"} Nov 04 04:47:41.830164 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.829864 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1ef6a31-f947-4b17-90ba-5d16957f9ea1","Type":"ContainerDied","Data":"28470ee02cff26b5794370760347764b1cc84a135e9df9eb6906727cdb431219"} Nov 04 04:47:41.836482 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.836466 4395 scope.go:122] "RemoveContainer" containerID="cb7838e8ec53d290b36e440deffbb2d765c09104248c224e7594d6b43cdd5bea" Nov 04 04:47:41.843142 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.843126 4395 scope.go:122] "RemoveContainer" containerID="fef92cb15f635b1ee9fe946cf12740dff3fd7b628be9b943d196e5909b057cb9" Nov 04 04:47:41.849424 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.849407 4395 scope.go:122] "RemoveContainer" containerID="e7a19405d75261ef50d459ad8359da6da340708cc670b049efd2ac7b7a328525" Nov 04 04:47:41.855439 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.855423 4395 scope.go:122] "RemoveContainer" containerID="50de7fa737b496c79942243e7cfdc8e0c58761e0b493506b8944e58d7009af81" Nov 04 04:47:41.861677 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.861663 4395 scope.go:122] "RemoveContainer" containerID="d4a4beba708596d4f6159dd0193793ea0dcc87e860cc968acfcc06a06540b538" Nov 04 04:47:41.868502 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.868486 4395 scope.go:122] "RemoveContainer" containerID="528ff97e2117bb6babeef289a032fdad24c21dc56c710085a27f08d2a83a2dfc" Nov 04 04:47:41.877012 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.876996 4395 scope.go:122] "RemoveContainer" containerID="71549b35337120be7ee09e530c88cec5725ddd1bbbb36cbb7f861be325217157" Nov 04 04:47:41.877244 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:47:41.877225 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71549b35337120be7ee09e530c88cec5725ddd1bbbb36cbb7f861be325217157\": container with ID starting with 71549b35337120be7ee09e530c88cec5725ddd1bbbb36cbb7f861be325217157 not found: ID does not exist" containerID="71549b35337120be7ee09e530c88cec5725ddd1bbbb36cbb7f861be325217157" Nov 04 04:47:41.877327 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.877258 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71549b35337120be7ee09e530c88cec5725ddd1bbbb36cbb7f861be325217157"} err="failed to get container status \"71549b35337120be7ee09e530c88cec5725ddd1bbbb36cbb7f861be325217157\": rpc error: code = NotFound desc = could not find container \"71549b35337120be7ee09e530c88cec5725ddd1bbbb36cbb7f861be325217157\": container with ID starting with 71549b35337120be7ee09e530c88cec5725ddd1bbbb36cbb7f861be325217157 not found: ID does not exist" Nov 04 04:47:41.877327 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.877280 4395 scope.go:122] "RemoveContainer" containerID="cb7838e8ec53d290b36e440deffbb2d765c09104248c224e7594d6b43cdd5bea" Nov 04 04:47:41.877584 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:47:41.877567 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb7838e8ec53d290b36e440deffbb2d765c09104248c224e7594d6b43cdd5bea\": container with ID starting with cb7838e8ec53d290b36e440deffbb2d765c09104248c224e7594d6b43cdd5bea not found: ID does not exist" containerID="cb7838e8ec53d290b36e440deffbb2d765c09104248c224e7594d6b43cdd5bea" Nov 04 04:47:41.877624 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.877591 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb7838e8ec53d290b36e440deffbb2d765c09104248c224e7594d6b43cdd5bea"} err="failed to get container status \"cb7838e8ec53d290b36e440deffbb2d765c09104248c224e7594d6b43cdd5bea\": rpc error: code = NotFound desc = could not find container \"cb7838e8ec53d290b36e440deffbb2d765c09104248c224e7594d6b43cdd5bea\": container with ID starting with cb7838e8ec53d290b36e440deffbb2d765c09104248c224e7594d6b43cdd5bea not found: ID does not exist" Nov 04 04:47:41.877624 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.877607 4395 scope.go:122] "RemoveContainer" containerID="fef92cb15f635b1ee9fe946cf12740dff3fd7b628be9b943d196e5909b057cb9" Nov 04 04:47:41.877854 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:47:41.877836 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fef92cb15f635b1ee9fe946cf12740dff3fd7b628be9b943d196e5909b057cb9\": container with ID starting with fef92cb15f635b1ee9fe946cf12740dff3fd7b628be9b943d196e5909b057cb9 not found: ID does not exist" containerID="fef92cb15f635b1ee9fe946cf12740dff3fd7b628be9b943d196e5909b057cb9" Nov 04 04:47:41.877895 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.877869 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fef92cb15f635b1ee9fe946cf12740dff3fd7b628be9b943d196e5909b057cb9"} err="failed to get container status \"fef92cb15f635b1ee9fe946cf12740dff3fd7b628be9b943d196e5909b057cb9\": rpc error: code = NotFound desc = could not find container \"fef92cb15f635b1ee9fe946cf12740dff3fd7b628be9b943d196e5909b057cb9\": container with ID starting with fef92cb15f635b1ee9fe946cf12740dff3fd7b628be9b943d196e5909b057cb9 not found: ID does not exist" Nov 04 04:47:41.877895 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.877883 4395 scope.go:122] "RemoveContainer" containerID="e7a19405d75261ef50d459ad8359da6da340708cc670b049efd2ac7b7a328525" Nov 04 04:47:41.878119 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:47:41.878102 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7a19405d75261ef50d459ad8359da6da340708cc670b049efd2ac7b7a328525\": container with ID starting with e7a19405d75261ef50d459ad8359da6da340708cc670b049efd2ac7b7a328525 not found: ID does not exist" containerID="e7a19405d75261ef50d459ad8359da6da340708cc670b049efd2ac7b7a328525" Nov 04 04:47:41.878186 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.878124 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7a19405d75261ef50d459ad8359da6da340708cc670b049efd2ac7b7a328525"} err="failed to get container status \"e7a19405d75261ef50d459ad8359da6da340708cc670b049efd2ac7b7a328525\": rpc error: code = NotFound desc = could not find container \"e7a19405d75261ef50d459ad8359da6da340708cc670b049efd2ac7b7a328525\": container with ID starting with e7a19405d75261ef50d459ad8359da6da340708cc670b049efd2ac7b7a328525 not found: ID does not exist" Nov 04 04:47:41.878186 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.878139 4395 scope.go:122] "RemoveContainer" containerID="50de7fa737b496c79942243e7cfdc8e0c58761e0b493506b8944e58d7009af81" Nov 04 04:47:41.878409 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:47:41.878387 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50de7fa737b496c79942243e7cfdc8e0c58761e0b493506b8944e58d7009af81\": container with ID starting with 50de7fa737b496c79942243e7cfdc8e0c58761e0b493506b8944e58d7009af81 not found: ID does not exist" containerID="50de7fa737b496c79942243e7cfdc8e0c58761e0b493506b8944e58d7009af81" Nov 04 04:47:41.878490 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.878419 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50de7fa737b496c79942243e7cfdc8e0c58761e0b493506b8944e58d7009af81"} err="failed to get container status \"50de7fa737b496c79942243e7cfdc8e0c58761e0b493506b8944e58d7009af81\": rpc error: code = NotFound desc = could not find container \"50de7fa737b496c79942243e7cfdc8e0c58761e0b493506b8944e58d7009af81\": container with ID starting with 50de7fa737b496c79942243e7cfdc8e0c58761e0b493506b8944e58d7009af81 not found: ID does not exist" Nov 04 04:47:41.878490 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.878439 4395 scope.go:122] "RemoveContainer" containerID="d4a4beba708596d4f6159dd0193793ea0dcc87e860cc968acfcc06a06540b538" Nov 04 04:47:41.878673 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:47:41.878658 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4a4beba708596d4f6159dd0193793ea0dcc87e860cc968acfcc06a06540b538\": container with ID starting with d4a4beba708596d4f6159dd0193793ea0dcc87e860cc968acfcc06a06540b538 not found: ID does not exist" containerID="d4a4beba708596d4f6159dd0193793ea0dcc87e860cc968acfcc06a06540b538" Nov 04 04:47:41.878715 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.878678 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a4beba708596d4f6159dd0193793ea0dcc87e860cc968acfcc06a06540b538"} err="failed to get container status \"d4a4beba708596d4f6159dd0193793ea0dcc87e860cc968acfcc06a06540b538\": rpc error: code = NotFound desc = could not find container \"d4a4beba708596d4f6159dd0193793ea0dcc87e860cc968acfcc06a06540b538\": container with ID starting with d4a4beba708596d4f6159dd0193793ea0dcc87e860cc968acfcc06a06540b538 not found: ID does not exist" Nov 04 04:47:41.878715 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.878692 4395 scope.go:122] "RemoveContainer" containerID="528ff97e2117bb6babeef289a032fdad24c21dc56c710085a27f08d2a83a2dfc" Nov 04 04:47:41.878886 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:47:41.878870 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"528ff97e2117bb6babeef289a032fdad24c21dc56c710085a27f08d2a83a2dfc\": container with ID starting with 528ff97e2117bb6babeef289a032fdad24c21dc56c710085a27f08d2a83a2dfc not found: ID does not exist" containerID="528ff97e2117bb6babeef289a032fdad24c21dc56c710085a27f08d2a83a2dfc" Nov 04 04:47:41.878929 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.878891 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"528ff97e2117bb6babeef289a032fdad24c21dc56c710085a27f08d2a83a2dfc"} err="failed to get container status \"528ff97e2117bb6babeef289a032fdad24c21dc56c710085a27f08d2a83a2dfc\": rpc error: code = NotFound desc = could not find container \"528ff97e2117bb6babeef289a032fdad24c21dc56c710085a27f08d2a83a2dfc\": container with ID starting with 528ff97e2117bb6babeef289a032fdad24c21dc56c710085a27f08d2a83a2dfc not found: ID does not exist" Nov 04 04:47:41.878929 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.878903 4395 scope.go:122] "RemoveContainer" containerID="71549b35337120be7ee09e530c88cec5725ddd1bbbb36cbb7f861be325217157" Nov 04 04:47:41.879087 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.879072 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71549b35337120be7ee09e530c88cec5725ddd1bbbb36cbb7f861be325217157"} err="failed to get container status \"71549b35337120be7ee09e530c88cec5725ddd1bbbb36cbb7f861be325217157\": rpc error: code = NotFound desc = could not find container \"71549b35337120be7ee09e530c88cec5725ddd1bbbb36cbb7f861be325217157\": container with ID starting with 71549b35337120be7ee09e530c88cec5725ddd1bbbb36cbb7f861be325217157 not found: ID does not exist" Nov 04 04:47:41.879121 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.879086 4395 scope.go:122] "RemoveContainer" containerID="cb7838e8ec53d290b36e440deffbb2d765c09104248c224e7594d6b43cdd5bea" Nov 04 04:47:41.879272 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.879257 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb7838e8ec53d290b36e440deffbb2d765c09104248c224e7594d6b43cdd5bea"} err="failed to get container status \"cb7838e8ec53d290b36e440deffbb2d765c09104248c224e7594d6b43cdd5bea\": rpc error: code = NotFound desc = could not find container \"cb7838e8ec53d290b36e440deffbb2d765c09104248c224e7594d6b43cdd5bea\": container with ID starting with cb7838e8ec53d290b36e440deffbb2d765c09104248c224e7594d6b43cdd5bea not found: ID does not exist" Nov 04 04:47:41.879337 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.879272 4395 scope.go:122] "RemoveContainer" containerID="fef92cb15f635b1ee9fe946cf12740dff3fd7b628be9b943d196e5909b057cb9" Nov 04 04:47:41.879487 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.879462 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fef92cb15f635b1ee9fe946cf12740dff3fd7b628be9b943d196e5909b057cb9"} err="failed to get container status \"fef92cb15f635b1ee9fe946cf12740dff3fd7b628be9b943d196e5909b057cb9\": rpc error: code = NotFound desc = could not find container \"fef92cb15f635b1ee9fe946cf12740dff3fd7b628be9b943d196e5909b057cb9\": container with ID starting with fef92cb15f635b1ee9fe946cf12740dff3fd7b628be9b943d196e5909b057cb9 not found: ID does not exist" Nov 04 04:47:41.879527 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.879488 4395 scope.go:122] "RemoveContainer" containerID="e7a19405d75261ef50d459ad8359da6da340708cc670b049efd2ac7b7a328525" Nov 04 04:47:41.879686 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.879669 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7a19405d75261ef50d459ad8359da6da340708cc670b049efd2ac7b7a328525"} err="failed to get container status \"e7a19405d75261ef50d459ad8359da6da340708cc670b049efd2ac7b7a328525\": rpc error: code = NotFound desc = could not find container \"e7a19405d75261ef50d459ad8359da6da340708cc670b049efd2ac7b7a328525\": container with ID starting with e7a19405d75261ef50d459ad8359da6da340708cc670b049efd2ac7b7a328525 not found: ID does not exist" Nov 04 04:47:41.879724 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.879687 4395 scope.go:122] "RemoveContainer" containerID="50de7fa737b496c79942243e7cfdc8e0c58761e0b493506b8944e58d7009af81" Nov 04 04:47:41.879904 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.879889 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50de7fa737b496c79942243e7cfdc8e0c58761e0b493506b8944e58d7009af81"} err="failed to get container status \"50de7fa737b496c79942243e7cfdc8e0c58761e0b493506b8944e58d7009af81\": rpc error: code = NotFound desc = could not find container \"50de7fa737b496c79942243e7cfdc8e0c58761e0b493506b8944e58d7009af81\": container with ID starting with 50de7fa737b496c79942243e7cfdc8e0c58761e0b493506b8944e58d7009af81 not found: ID does not exist" Nov 04 04:47:41.879984 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.879904 4395 scope.go:122] "RemoveContainer" containerID="d4a4beba708596d4f6159dd0193793ea0dcc87e860cc968acfcc06a06540b538" Nov 04 04:47:41.880125 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.880098 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a4beba708596d4f6159dd0193793ea0dcc87e860cc968acfcc06a06540b538"} err="failed to get container status \"d4a4beba708596d4f6159dd0193793ea0dcc87e860cc968acfcc06a06540b538\": rpc error: code = NotFound desc = could not find container \"d4a4beba708596d4f6159dd0193793ea0dcc87e860cc968acfcc06a06540b538\": container with ID starting with d4a4beba708596d4f6159dd0193793ea0dcc87e860cc968acfcc06a06540b538 not found: ID does not exist" Nov 04 04:47:41.880166 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.880125 4395 scope.go:122] "RemoveContainer" containerID="528ff97e2117bb6babeef289a032fdad24c21dc56c710085a27f08d2a83a2dfc" Nov 04 04:47:41.880341 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.880325 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"528ff97e2117bb6babeef289a032fdad24c21dc56c710085a27f08d2a83a2dfc"} err="failed to get container status \"528ff97e2117bb6babeef289a032fdad24c21dc56c710085a27f08d2a83a2dfc\": rpc error: code = NotFound desc = could not find container \"528ff97e2117bb6babeef289a032fdad24c21dc56c710085a27f08d2a83a2dfc\": container with ID starting with 528ff97e2117bb6babeef289a032fdad24c21dc56c710085a27f08d2a83a2dfc not found: ID does not exist" Nov 04 04:47:41.880389 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.880342 4395 scope.go:122] "RemoveContainer" containerID="71549b35337120be7ee09e530c88cec5725ddd1bbbb36cbb7f861be325217157" Nov 04 04:47:41.880529 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.880515 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71549b35337120be7ee09e530c88cec5725ddd1bbbb36cbb7f861be325217157"} err="failed to get container status \"71549b35337120be7ee09e530c88cec5725ddd1bbbb36cbb7f861be325217157\": rpc error: code = NotFound desc = could not find container \"71549b35337120be7ee09e530c88cec5725ddd1bbbb36cbb7f861be325217157\": container with ID starting with 71549b35337120be7ee09e530c88cec5725ddd1bbbb36cbb7f861be325217157 not found: ID does not exist" Nov 04 04:47:41.880568 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.880529 4395 scope.go:122] "RemoveContainer" containerID="cb7838e8ec53d290b36e440deffbb2d765c09104248c224e7594d6b43cdd5bea" Nov 04 04:47:41.880719 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.880701 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb7838e8ec53d290b36e440deffbb2d765c09104248c224e7594d6b43cdd5bea"} err="failed to get container status \"cb7838e8ec53d290b36e440deffbb2d765c09104248c224e7594d6b43cdd5bea\": rpc error: code = NotFound desc = could not find container \"cb7838e8ec53d290b36e440deffbb2d765c09104248c224e7594d6b43cdd5bea\": container with ID starting with cb7838e8ec53d290b36e440deffbb2d765c09104248c224e7594d6b43cdd5bea not found: ID does not exist" Nov 04 04:47:41.880766 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.880719 4395 scope.go:122] "RemoveContainer" containerID="fef92cb15f635b1ee9fe946cf12740dff3fd7b628be9b943d196e5909b057cb9" Nov 04 04:47:41.880927 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.880910 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fef92cb15f635b1ee9fe946cf12740dff3fd7b628be9b943d196e5909b057cb9"} err="failed to get container status \"fef92cb15f635b1ee9fe946cf12740dff3fd7b628be9b943d196e5909b057cb9\": rpc error: code = NotFound desc = could not find container \"fef92cb15f635b1ee9fe946cf12740dff3fd7b628be9b943d196e5909b057cb9\": container with ID starting with fef92cb15f635b1ee9fe946cf12740dff3fd7b628be9b943d196e5909b057cb9 not found: ID does not exist" Nov 04 04:47:41.880971 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.880928 4395 scope.go:122] "RemoveContainer" containerID="e7a19405d75261ef50d459ad8359da6da340708cc670b049efd2ac7b7a328525" Nov 04 04:47:41.881119 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.881102 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7a19405d75261ef50d459ad8359da6da340708cc670b049efd2ac7b7a328525"} err="failed to get container status \"e7a19405d75261ef50d459ad8359da6da340708cc670b049efd2ac7b7a328525\": rpc error: code = NotFound desc = could not find container \"e7a19405d75261ef50d459ad8359da6da340708cc670b049efd2ac7b7a328525\": container with ID starting with e7a19405d75261ef50d459ad8359da6da340708cc670b049efd2ac7b7a328525 not found: ID does not exist" Nov 04 04:47:41.881159 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.881119 4395 scope.go:122] "RemoveContainer" containerID="50de7fa737b496c79942243e7cfdc8e0c58761e0b493506b8944e58d7009af81" Nov 04 04:47:41.881333 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.881311 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50de7fa737b496c79942243e7cfdc8e0c58761e0b493506b8944e58d7009af81"} err="failed to get container status \"50de7fa737b496c79942243e7cfdc8e0c58761e0b493506b8944e58d7009af81\": rpc error: code = NotFound desc = could not find container \"50de7fa737b496c79942243e7cfdc8e0c58761e0b493506b8944e58d7009af81\": container with ID starting with 50de7fa737b496c79942243e7cfdc8e0c58761e0b493506b8944e58d7009af81 not found: ID does not exist" Nov 04 04:47:41.881408 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.881335 4395 scope.go:122] "RemoveContainer" containerID="d4a4beba708596d4f6159dd0193793ea0dcc87e860cc968acfcc06a06540b538" Nov 04 04:47:41.881537 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.881522 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a4beba708596d4f6159dd0193793ea0dcc87e860cc968acfcc06a06540b538"} err="failed to get container status \"d4a4beba708596d4f6159dd0193793ea0dcc87e860cc968acfcc06a06540b538\": rpc error: code = NotFound desc = could not find container \"d4a4beba708596d4f6159dd0193793ea0dcc87e860cc968acfcc06a06540b538\": container with ID starting with d4a4beba708596d4f6159dd0193793ea0dcc87e860cc968acfcc06a06540b538 not found: ID does not exist" Nov 04 04:47:41.881586 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.881539 4395 scope.go:122] "RemoveContainer" containerID="528ff97e2117bb6babeef289a032fdad24c21dc56c710085a27f08d2a83a2dfc" Nov 04 04:47:41.881715 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.881695 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"528ff97e2117bb6babeef289a032fdad24c21dc56c710085a27f08d2a83a2dfc"} err="failed to get container status \"528ff97e2117bb6babeef289a032fdad24c21dc56c710085a27f08d2a83a2dfc\": rpc error: code = NotFound desc = could not find container \"528ff97e2117bb6babeef289a032fdad24c21dc56c710085a27f08d2a83a2dfc\": container with ID starting with 528ff97e2117bb6babeef289a032fdad24c21dc56c710085a27f08d2a83a2dfc not found: ID does not exist" Nov 04 04:47:41.881755 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.881716 4395 scope.go:122] "RemoveContainer" containerID="71549b35337120be7ee09e530c88cec5725ddd1bbbb36cbb7f861be325217157" Nov 04 04:47:41.881892 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.881876 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71549b35337120be7ee09e530c88cec5725ddd1bbbb36cbb7f861be325217157"} err="failed to get container status \"71549b35337120be7ee09e530c88cec5725ddd1bbbb36cbb7f861be325217157\": rpc error: code = NotFound desc = could not find container \"71549b35337120be7ee09e530c88cec5725ddd1bbbb36cbb7f861be325217157\": container with ID starting with 71549b35337120be7ee09e530c88cec5725ddd1bbbb36cbb7f861be325217157 not found: ID does not exist" Nov 04 04:47:41.881932 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.881891 4395 scope.go:122] "RemoveContainer" containerID="cb7838e8ec53d290b36e440deffbb2d765c09104248c224e7594d6b43cdd5bea" Nov 04 04:47:41.882073 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.882053 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb7838e8ec53d290b36e440deffbb2d765c09104248c224e7594d6b43cdd5bea"} err="failed to get container status \"cb7838e8ec53d290b36e440deffbb2d765c09104248c224e7594d6b43cdd5bea\": rpc error: code = NotFound desc = could not find container \"cb7838e8ec53d290b36e440deffbb2d765c09104248c224e7594d6b43cdd5bea\": container with ID starting with cb7838e8ec53d290b36e440deffbb2d765c09104248c224e7594d6b43cdd5bea not found: ID does not exist" Nov 04 04:47:41.882116 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.882073 4395 scope.go:122] "RemoveContainer" containerID="fef92cb15f635b1ee9fe946cf12740dff3fd7b628be9b943d196e5909b057cb9" Nov 04 04:47:41.882239 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.882225 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fef92cb15f635b1ee9fe946cf12740dff3fd7b628be9b943d196e5909b057cb9"} err="failed to get container status \"fef92cb15f635b1ee9fe946cf12740dff3fd7b628be9b943d196e5909b057cb9\": rpc error: code = NotFound desc = could not find container \"fef92cb15f635b1ee9fe946cf12740dff3fd7b628be9b943d196e5909b057cb9\": container with ID starting with fef92cb15f635b1ee9fe946cf12740dff3fd7b628be9b943d196e5909b057cb9 not found: ID does not exist" Nov 04 04:47:41.882280 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.882238 4395 scope.go:122] "RemoveContainer" containerID="e7a19405d75261ef50d459ad8359da6da340708cc670b049efd2ac7b7a328525" Nov 04 04:47:41.882412 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.882397 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7a19405d75261ef50d459ad8359da6da340708cc670b049efd2ac7b7a328525"} err="failed to get container status \"e7a19405d75261ef50d459ad8359da6da340708cc670b049efd2ac7b7a328525\": rpc error: code = NotFound desc = could not find container \"e7a19405d75261ef50d459ad8359da6da340708cc670b049efd2ac7b7a328525\": container with ID starting with e7a19405d75261ef50d459ad8359da6da340708cc670b049efd2ac7b7a328525 not found: ID does not exist" Nov 04 04:47:41.882456 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.882412 4395 scope.go:122] "RemoveContainer" containerID="50de7fa737b496c79942243e7cfdc8e0c58761e0b493506b8944e58d7009af81" Nov 04 04:47:41.882586 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.882572 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50de7fa737b496c79942243e7cfdc8e0c58761e0b493506b8944e58d7009af81"} err="failed to get container status \"50de7fa737b496c79942243e7cfdc8e0c58761e0b493506b8944e58d7009af81\": rpc error: code = NotFound desc = could not find container \"50de7fa737b496c79942243e7cfdc8e0c58761e0b493506b8944e58d7009af81\": container with ID starting with 50de7fa737b496c79942243e7cfdc8e0c58761e0b493506b8944e58d7009af81 not found: ID does not exist" Nov 04 04:47:41.882630 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.882586 4395 scope.go:122] "RemoveContainer" containerID="d4a4beba708596d4f6159dd0193793ea0dcc87e860cc968acfcc06a06540b538" Nov 04 04:47:41.882765 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.882745 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a4beba708596d4f6159dd0193793ea0dcc87e860cc968acfcc06a06540b538"} err="failed to get container status \"d4a4beba708596d4f6159dd0193793ea0dcc87e860cc968acfcc06a06540b538\": rpc error: code = NotFound desc = could not find container \"d4a4beba708596d4f6159dd0193793ea0dcc87e860cc968acfcc06a06540b538\": container with ID starting with d4a4beba708596d4f6159dd0193793ea0dcc87e860cc968acfcc06a06540b538 not found: ID does not exist" Nov 04 04:47:41.882802 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.882765 4395 scope.go:122] "RemoveContainer" containerID="528ff97e2117bb6babeef289a032fdad24c21dc56c710085a27f08d2a83a2dfc" Nov 04 04:47:41.882937 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.882918 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"528ff97e2117bb6babeef289a032fdad24c21dc56c710085a27f08d2a83a2dfc"} err="failed to get container status \"528ff97e2117bb6babeef289a032fdad24c21dc56c710085a27f08d2a83a2dfc\": rpc error: code = NotFound desc = could not find container \"528ff97e2117bb6babeef289a032fdad24c21dc56c710085a27f08d2a83a2dfc\": container with ID starting with 528ff97e2117bb6babeef289a032fdad24c21dc56c710085a27f08d2a83a2dfc not found: ID does not exist" Nov 04 04:47:41.882973 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.882938 4395 scope.go:122] "RemoveContainer" containerID="71549b35337120be7ee09e530c88cec5725ddd1bbbb36cbb7f861be325217157" Nov 04 04:47:41.883109 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.883095 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71549b35337120be7ee09e530c88cec5725ddd1bbbb36cbb7f861be325217157"} err="failed to get container status \"71549b35337120be7ee09e530c88cec5725ddd1bbbb36cbb7f861be325217157\": rpc error: code = NotFound desc = could not find container \"71549b35337120be7ee09e530c88cec5725ddd1bbbb36cbb7f861be325217157\": container with ID starting with 71549b35337120be7ee09e530c88cec5725ddd1bbbb36cbb7f861be325217157 not found: ID does not exist" Nov 04 04:47:41.883109 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.883108 4395 scope.go:122] "RemoveContainer" containerID="cb7838e8ec53d290b36e440deffbb2d765c09104248c224e7594d6b43cdd5bea" Nov 04 04:47:41.883290 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.883271 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb7838e8ec53d290b36e440deffbb2d765c09104248c224e7594d6b43cdd5bea"} err="failed to get container status \"cb7838e8ec53d290b36e440deffbb2d765c09104248c224e7594d6b43cdd5bea\": rpc error: code = NotFound desc = could not find container \"cb7838e8ec53d290b36e440deffbb2d765c09104248c224e7594d6b43cdd5bea\": container with ID starting with cb7838e8ec53d290b36e440deffbb2d765c09104248c224e7594d6b43cdd5bea not found: ID does not exist" Nov 04 04:47:41.883342 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.883290 4395 scope.go:122] "RemoveContainer" containerID="fef92cb15f635b1ee9fe946cf12740dff3fd7b628be9b943d196e5909b057cb9" Nov 04 04:47:41.883500 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.883483 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fef92cb15f635b1ee9fe946cf12740dff3fd7b628be9b943d196e5909b057cb9"} err="failed to get container status \"fef92cb15f635b1ee9fe946cf12740dff3fd7b628be9b943d196e5909b057cb9\": rpc error: code = NotFound desc = could not find container \"fef92cb15f635b1ee9fe946cf12740dff3fd7b628be9b943d196e5909b057cb9\": container with ID starting with fef92cb15f635b1ee9fe946cf12740dff3fd7b628be9b943d196e5909b057cb9 not found: ID does not exist" Nov 04 04:47:41.883568 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.883501 4395 scope.go:122] "RemoveContainer" containerID="e7a19405d75261ef50d459ad8359da6da340708cc670b049efd2ac7b7a328525" Nov 04 04:47:41.883733 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.883719 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7a19405d75261ef50d459ad8359da6da340708cc670b049efd2ac7b7a328525"} err="failed to get container status \"e7a19405d75261ef50d459ad8359da6da340708cc670b049efd2ac7b7a328525\": rpc error: code = NotFound desc = could not find container \"e7a19405d75261ef50d459ad8359da6da340708cc670b049efd2ac7b7a328525\": container with ID starting with e7a19405d75261ef50d459ad8359da6da340708cc670b049efd2ac7b7a328525 not found: ID does not exist" Nov 04 04:47:41.883777 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.883732 4395 scope.go:122] "RemoveContainer" containerID="50de7fa737b496c79942243e7cfdc8e0c58761e0b493506b8944e58d7009af81" Nov 04 04:47:41.883909 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.883892 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50de7fa737b496c79942243e7cfdc8e0c58761e0b493506b8944e58d7009af81"} err="failed to get container status \"50de7fa737b496c79942243e7cfdc8e0c58761e0b493506b8944e58d7009af81\": rpc error: code = NotFound desc = could not find container \"50de7fa737b496c79942243e7cfdc8e0c58761e0b493506b8944e58d7009af81\": container with ID starting with 50de7fa737b496c79942243e7cfdc8e0c58761e0b493506b8944e58d7009af81 not found: ID does not exist" Nov 04 04:47:41.883974 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.883910 4395 scope.go:122] "RemoveContainer" containerID="d4a4beba708596d4f6159dd0193793ea0dcc87e860cc968acfcc06a06540b538" Nov 04 04:47:41.884127 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.884111 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a4beba708596d4f6159dd0193793ea0dcc87e860cc968acfcc06a06540b538"} err="failed to get container status \"d4a4beba708596d4f6159dd0193793ea0dcc87e860cc968acfcc06a06540b538\": rpc error: code = NotFound desc = could not find container \"d4a4beba708596d4f6159dd0193793ea0dcc87e860cc968acfcc06a06540b538\": container with ID starting with d4a4beba708596d4f6159dd0193793ea0dcc87e860cc968acfcc06a06540b538 not found: ID does not exist" Nov 04 04:47:41.884127 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.884126 4395 scope.go:122] "RemoveContainer" containerID="528ff97e2117bb6babeef289a032fdad24c21dc56c710085a27f08d2a83a2dfc" Nov 04 04:47:41.884328 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.884290 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"528ff97e2117bb6babeef289a032fdad24c21dc56c710085a27f08d2a83a2dfc"} err="failed to get container status \"528ff97e2117bb6babeef289a032fdad24c21dc56c710085a27f08d2a83a2dfc\": rpc error: code = NotFound desc = could not find container \"528ff97e2117bb6babeef289a032fdad24c21dc56c710085a27f08d2a83a2dfc\": container with ID starting with 528ff97e2117bb6babeef289a032fdad24c21dc56c710085a27f08d2a83a2dfc not found: ID does not exist" Nov 04 04:47:41.884378 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.884329 4395 scope.go:122] "RemoveContainer" containerID="71549b35337120be7ee09e530c88cec5725ddd1bbbb36cbb7f861be325217157" Nov 04 04:47:41.884505 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.884490 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71549b35337120be7ee09e530c88cec5725ddd1bbbb36cbb7f861be325217157"} err="failed to get container status \"71549b35337120be7ee09e530c88cec5725ddd1bbbb36cbb7f861be325217157\": rpc error: code = NotFound desc = could not find container \"71549b35337120be7ee09e530c88cec5725ddd1bbbb36cbb7f861be325217157\": container with ID starting with 71549b35337120be7ee09e530c88cec5725ddd1bbbb36cbb7f861be325217157 not found: ID does not exist" Nov 04 04:47:41.884505 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.884504 4395 scope.go:122] "RemoveContainer" containerID="cb7838e8ec53d290b36e440deffbb2d765c09104248c224e7594d6b43cdd5bea" Nov 04 04:47:41.884687 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.884669 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb7838e8ec53d290b36e440deffbb2d765c09104248c224e7594d6b43cdd5bea"} err="failed to get container status \"cb7838e8ec53d290b36e440deffbb2d765c09104248c224e7594d6b43cdd5bea\": rpc error: code = NotFound desc = could not find container \"cb7838e8ec53d290b36e440deffbb2d765c09104248c224e7594d6b43cdd5bea\": container with ID starting with cb7838e8ec53d290b36e440deffbb2d765c09104248c224e7594d6b43cdd5bea not found: ID does not exist" Nov 04 04:47:41.884754 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.884688 4395 scope.go:122] "RemoveContainer" containerID="fef92cb15f635b1ee9fe946cf12740dff3fd7b628be9b943d196e5909b057cb9" Nov 04 04:47:41.884877 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.884860 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fef92cb15f635b1ee9fe946cf12740dff3fd7b628be9b943d196e5909b057cb9"} err="failed to get container status \"fef92cb15f635b1ee9fe946cf12740dff3fd7b628be9b943d196e5909b057cb9\": rpc error: code = NotFound desc = could not find container \"fef92cb15f635b1ee9fe946cf12740dff3fd7b628be9b943d196e5909b057cb9\": container with ID starting with fef92cb15f635b1ee9fe946cf12740dff3fd7b628be9b943d196e5909b057cb9 not found: ID does not exist" Nov 04 04:47:41.884923 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.884877 4395 scope.go:122] "RemoveContainer" containerID="e7a19405d75261ef50d459ad8359da6da340708cc670b049efd2ac7b7a328525" Nov 04 04:47:41.885077 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.885059 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7a19405d75261ef50d459ad8359da6da340708cc670b049efd2ac7b7a328525"} err="failed to get container status \"e7a19405d75261ef50d459ad8359da6da340708cc670b049efd2ac7b7a328525\": rpc error: code = NotFound desc = could not find container \"e7a19405d75261ef50d459ad8359da6da340708cc670b049efd2ac7b7a328525\": container with ID starting with e7a19405d75261ef50d459ad8359da6da340708cc670b049efd2ac7b7a328525 not found: ID does not exist" Nov 04 04:47:41.885142 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.885078 4395 scope.go:122] "RemoveContainer" containerID="50de7fa737b496c79942243e7cfdc8e0c58761e0b493506b8944e58d7009af81" Nov 04 04:47:41.885280 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.885262 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50de7fa737b496c79942243e7cfdc8e0c58761e0b493506b8944e58d7009af81"} err="failed to get container status \"50de7fa737b496c79942243e7cfdc8e0c58761e0b493506b8944e58d7009af81\": rpc error: code = NotFound desc = could not find container \"50de7fa737b496c79942243e7cfdc8e0c58761e0b493506b8944e58d7009af81\": container with ID starting with 50de7fa737b496c79942243e7cfdc8e0c58761e0b493506b8944e58d7009af81 not found: ID does not exist" Nov 04 04:47:41.885345 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.885279 4395 scope.go:122] "RemoveContainer" containerID="d4a4beba708596d4f6159dd0193793ea0dcc87e860cc968acfcc06a06540b538" Nov 04 04:47:41.885496 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.885478 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a4beba708596d4f6159dd0193793ea0dcc87e860cc968acfcc06a06540b538"} err="failed to get container status \"d4a4beba708596d4f6159dd0193793ea0dcc87e860cc968acfcc06a06540b538\": rpc error: code = NotFound desc = could not find container \"d4a4beba708596d4f6159dd0193793ea0dcc87e860cc968acfcc06a06540b538\": container with ID starting with d4a4beba708596d4f6159dd0193793ea0dcc87e860cc968acfcc06a06540b538 not found: ID does not exist" Nov 04 04:47:41.885562 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.885496 4395 scope.go:122] "RemoveContainer" containerID="528ff97e2117bb6babeef289a032fdad24c21dc56c710085a27f08d2a83a2dfc" Nov 04 04:47:41.885686 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.885671 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"528ff97e2117bb6babeef289a032fdad24c21dc56c710085a27f08d2a83a2dfc"} err="failed to get container status \"528ff97e2117bb6babeef289a032fdad24c21dc56c710085a27f08d2a83a2dfc\": rpc error: code = NotFound desc = could not find container \"528ff97e2117bb6babeef289a032fdad24c21dc56c710085a27f08d2a83a2dfc\": container with ID starting with 528ff97e2117bb6babeef289a032fdad24c21dc56c710085a27f08d2a83a2dfc not found: ID does not exist" Nov 04 04:47:41.912943 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.912920 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-prometheus-k8s-rulefiles-0\") pod \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " Nov 04 04:47:41.913040 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.912948 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " Nov 04 04:47:41.913040 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.912979 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-config\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-config\") pod \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " Nov 04 04:47:41.913040 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.913010 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-prometheus-k8s-rulefiles-2\") pod \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " Nov 04 04:47:41.913197 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.913039 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-prometheus-trusted-ca-bundle\") pod \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " Nov 04 04:47:41.913197 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.913077 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-config-out\" (UniqueName: \"kubernetes.io/empty-dir/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-config-out\") pod \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " Nov 04 04:47:41.913197 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.913105 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-thanos-prometheus-http-client-file\") pod \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " Nov 04 04:47:41.913197 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.913137 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-web-config\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-web-config\") pod \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " Nov 04 04:47:41.913197 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.913170 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-grpc-tls\") pod \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " Nov 04 04:47:41.913462 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.913226 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-kube-api-access-952s7\" (UniqueName: \"kubernetes.io/projected/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-kube-api-access-952s7\") pod \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " Nov 04 04:47:41.913462 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.913261 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-configmap-serving-certs-ca-bundle\") pod \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " Nov 04 04:47:41.913462 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.913348 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-configmap-metrics-client-ca\") pod \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " Nov 04 04:47:41.913462 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.913385 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-prometheus-k8s-tls\") pod \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " Nov 04 04:47:41.913462 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.913425 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-prometheus-k8s-rulefiles-1\") pod \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " Nov 04 04:47:41.913713 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.913468 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-tls-assets\" (UniqueName: \"kubernetes.io/projected/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-tls-assets\") pod \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " Nov 04 04:47:41.913713 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.913498 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-metrics-client-certs\") pod \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " Nov 04 04:47:41.913713 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.913541 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " Nov 04 04:47:41.913713 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.913572 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-kube-rbac-proxy\") pod \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " Nov 04 04:47:41.913713 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.913601 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-configmap-kubelet-serving-ca-bundle\") pod \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " Nov 04 04:47:41.913713 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.913633 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-prometheus-k8s-db\") pod \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\" (UID: \"a1ef6a31-f947-4b17-90ba-5d16957f9ea1\") " Nov 04 04:47:41.915270 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.913382 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-prometheus-k8s-rulefiles-2" pod "a1ef6a31-f947-4b17-90ba-5d16957f9ea1" (UID: "a1ef6a31-f947-4b17-90ba-5d16957f9ea1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:47:41.915270 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.913483 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-prometheus-trusted-ca-bundle" pod "a1ef6a31-f947-4b17-90ba-5d16957f9ea1" (UID: "a1ef6a31-f947-4b17-90ba-5d16957f9ea1"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:47:41.915270 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.913917 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-configmap-metrics-client-ca" pod "a1ef6a31-f947-4b17-90ba-5d16957f9ea1" (UID: "a1ef6a31-f947-4b17-90ba-5d16957f9ea1"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:47:41.915270 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.914687 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-prometheus-k8s-rulefiles-0" pod "a1ef6a31-f947-4b17-90ba-5d16957f9ea1" (UID: "a1ef6a31-f947-4b17-90ba-5d16957f9ea1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:47:41.915270 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.915195 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-prometheus-k8s-db" pod "a1ef6a31-f947-4b17-90ba-5d16957f9ea1" (UID: "a1ef6a31-f947-4b17-90ba-5d16957f9ea1"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:47:41.915713 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.915673 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-thanos-prometheus-http-client-file" pod "a1ef6a31-f947-4b17-90ba-5d16957f9ea1" (UID: "a1ef6a31-f947-4b17-90ba-5d16957f9ea1"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:47:41.916238 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.916022 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-prometheus-k8s-rulefiles-1" pod "a1ef6a31-f947-4b17-90ba-5d16957f9ea1" (UID: "a1ef6a31-f947-4b17-90ba-5d16957f9ea1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:47:41.916238 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.916037 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-config" pod "a1ef6a31-f947-4b17-90ba-5d16957f9ea1" (UID: "a1ef6a31-f947-4b17-90ba-5d16957f9ea1"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:47:41.916238 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.916099 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-prometheus-k8s-kube-rbac-proxy-web" pod "a1ef6a31-f947-4b17-90ba-5d16957f9ea1" (UID: "a1ef6a31-f947-4b17-90ba-5d16957f9ea1"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:47:41.916788 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.916740 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-configmap-kubelet-serving-ca-bundle" pod "a1ef6a31-f947-4b17-90ba-5d16957f9ea1" (UID: "a1ef6a31-f947-4b17-90ba-5d16957f9ea1"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:47:41.916788 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.916755 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-configmap-serving-certs-ca-bundle" pod "a1ef6a31-f947-4b17-90ba-5d16957f9ea1" (UID: "a1ef6a31-f947-4b17-90ba-5d16957f9ea1"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:47:41.916931 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.916812 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-grpc-tls" pod "a1ef6a31-f947-4b17-90ba-5d16957f9ea1" (UID: "a1ef6a31-f947-4b17-90ba-5d16957f9ea1"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:47:41.917376 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.917344 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-config-out" pod "a1ef6a31-f947-4b17-90ba-5d16957f9ea1" (UID: "a1ef6a31-f947-4b17-90ba-5d16957f9ea1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:47:41.917376 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.917362 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-tls-assets" pod "a1ef6a31-f947-4b17-90ba-5d16957f9ea1" (UID: "a1ef6a31-f947-4b17-90ba-5d16957f9ea1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:47:41.917631 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.917609 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-metrics-client-certs" pod "a1ef6a31-f947-4b17-90ba-5d16957f9ea1" (UID: "a1ef6a31-f947-4b17-90ba-5d16957f9ea1"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:47:41.917964 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.917942 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-kube-api-access-952s7" pod "a1ef6a31-f947-4b17-90ba-5d16957f9ea1" (UID: "a1ef6a31-f947-4b17-90ba-5d16957f9ea1"). InnerVolumeSpecName "kube-api-access-952s7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:47:41.918442 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.918424 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-prometheus-k8s-thanos-sidecar-tls" pod "a1ef6a31-f947-4b17-90ba-5d16957f9ea1" (UID: "a1ef6a31-f947-4b17-90ba-5d16957f9ea1"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:47:41.918814 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.918792 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-kube-rbac-proxy" pod "a1ef6a31-f947-4b17-90ba-5d16957f9ea1" (UID: "a1ef6a31-f947-4b17-90ba-5d16957f9ea1"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:47:41.919118 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.919099 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-prometheus-k8s-tls" pod "a1ef6a31-f947-4b17-90ba-5d16957f9ea1" (UID: "a1ef6a31-f947-4b17-90ba-5d16957f9ea1"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:47:41.926666 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:41.926648 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-web-config" pod "a1ef6a31-f947-4b17-90ba-5d16957f9ea1" (UID: "a1ef6a31-f947-4b17-90ba-5d16957f9ea1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:47:42.015034 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.015008 4395 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-grpc-tls\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:47:42.015034 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.015033 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-952s7\" (UniqueName: \"kubernetes.io/projected/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-kube-api-access-952s7\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:47:42.015192 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.015042 4395 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:47:42.015192 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.015052 4395 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-configmap-metrics-client-ca\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:47:42.015192 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.015061 4395 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-prometheus-k8s-tls\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:47:42.015192 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.015069 4395 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:47:42.015192 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.015079 4395 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-tls-assets\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:47:42.015192 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.015089 4395 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-metrics-client-certs\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:47:42.015192 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.015097 4395 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:47:42.015192 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.015106 4395 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-kube-rbac-proxy\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:47:42.015192 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.015114 4395 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:47:42.015192 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.015123 4395 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-prometheus-k8s-db\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:47:42.015192 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.015131 4395 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:47:42.015192 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.015139 4395 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:47:42.015192 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.015149 4395 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-config\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:47:42.015192 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.015157 4395 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:47:42.015192 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.015166 4395 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-prometheus-trusted-ca-bundle\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:47:42.015192 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.015175 4395 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-config-out\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:47:42.015192 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.015183 4395 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:47:42.015192 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.015191 4395 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a1ef6a31-f947-4b17-90ba-5d16957f9ea1-web-config\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:47:42.243882 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.243853 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 04:47:42.247756 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.247706 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 04:47:42.277383 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.277357 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 04:47:42.277610 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.277598 4395 cpu_manager.go:414] "containerMap: removing container" podUID="a1ef6a31-f947-4b17-90ba-5d16957f9ea1" containerName="kube-rbac-proxy-thanos" Nov 04 04:47:42.277651 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.277611 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1ef6a31-f947-4b17-90ba-5d16957f9ea1" containerName="kube-rbac-proxy-thanos" Nov 04 04:47:42.277651 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.277627 4395 cpu_manager.go:414] "containerMap: removing container" podUID="abe56d25-e504-4167-8a70-95cc1918b64f" containerName="console" Nov 04 04:47:42.277651 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.277632 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abe56d25-e504-4167-8a70-95cc1918b64f" containerName="console" Nov 04 04:47:42.277651 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.277641 4395 cpu_manager.go:414] "containerMap: removing container" podUID="a1ef6a31-f947-4b17-90ba-5d16957f9ea1" containerName="config-reloader" Nov 04 04:47:42.277651 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.277646 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1ef6a31-f947-4b17-90ba-5d16957f9ea1" containerName="config-reloader" Nov 04 04:47:42.277792 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.277656 4395 cpu_manager.go:414] "containerMap: removing container" podUID="a1ef6a31-f947-4b17-90ba-5d16957f9ea1" containerName="thanos-sidecar" Nov 04 04:47:42.277792 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.277661 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1ef6a31-f947-4b17-90ba-5d16957f9ea1" containerName="thanos-sidecar" Nov 04 04:47:42.277792 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.277668 4395 cpu_manager.go:414] "containerMap: removing container" podUID="a1ef6a31-f947-4b17-90ba-5d16957f9ea1" containerName="kube-rbac-proxy-web" Nov 04 04:47:42.277792 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.277672 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1ef6a31-f947-4b17-90ba-5d16957f9ea1" containerName="kube-rbac-proxy-web" Nov 04 04:47:42.277792 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.277679 4395 cpu_manager.go:414] "containerMap: removing container" podUID="a1ef6a31-f947-4b17-90ba-5d16957f9ea1" containerName="kube-rbac-proxy" Nov 04 04:47:42.277792 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.277684 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1ef6a31-f947-4b17-90ba-5d16957f9ea1" containerName="kube-rbac-proxy" Nov 04 04:47:42.277792 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.277694 4395 cpu_manager.go:414] "containerMap: removing container" podUID="a1ef6a31-f947-4b17-90ba-5d16957f9ea1" containerName="init-config-reloader" Nov 04 04:47:42.277792 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.277699 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1ef6a31-f947-4b17-90ba-5d16957f9ea1" containerName="init-config-reloader" Nov 04 04:47:42.277792 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.277705 4395 cpu_manager.go:414] "containerMap: removing container" podUID="a1ef6a31-f947-4b17-90ba-5d16957f9ea1" containerName="prometheus" Nov 04 04:47:42.277792 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.277709 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1ef6a31-f947-4b17-90ba-5d16957f9ea1" containerName="prometheus" Nov 04 04:47:42.277792 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.277747 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1ef6a31-f947-4b17-90ba-5d16957f9ea1" containerName="thanos-sidecar" Nov 04 04:47:42.277792 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.277755 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1ef6a31-f947-4b17-90ba-5d16957f9ea1" containerName="kube-rbac-proxy-web" Nov 04 04:47:42.277792 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.277760 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1ef6a31-f947-4b17-90ba-5d16957f9ea1" containerName="kube-rbac-proxy" Nov 04 04:47:42.277792 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.277766 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1ef6a31-f947-4b17-90ba-5d16957f9ea1" containerName="prometheus" Nov 04 04:47:42.277792 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.277772 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1ef6a31-f947-4b17-90ba-5d16957f9ea1" containerName="kube-rbac-proxy-thanos" Nov 04 04:47:42.277792 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.277780 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="abe56d25-e504-4167-8a70-95cc1918b64f" containerName="console" Nov 04 04:47:42.277792 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.277790 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1ef6a31-f947-4b17-90ba-5d16957f9ea1" containerName="config-reloader" Nov 04 04:47:42.282345 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.282330 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.331941 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.331916 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 04 04:47:42.336972 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.332865 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 04 04:47:42.336972 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.333117 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 04 04:47:42.336972 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.333585 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 04 04:47:42.336972 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.333595 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 04 04:47:42.336972 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.333822 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 04 04:47:42.336972 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.333832 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 04 04:47:42.336972 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.333828 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 04 04:47:42.336972 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.333983 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 04 04:47:42.336972 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.334045 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 04 04:47:42.336972 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.334072 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 04 04:47:42.336972 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.334223 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 04 04:47:42.336972 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.334486 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2cm5h\"" Nov 04 04:47:42.336972 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.334535 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 04 04:47:42.379498 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.379477 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ca9nc2q72skrd\"" Nov 04 04:47:42.379614 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.379552 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 04 04:47:42.379614 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.379555 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 04 04:47:42.380317 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.380286 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 04:47:42.413484 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.413417 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1ef6a31-f947-4b17-90ba-5d16957f9ea1" path="/var/lib/kubelet/pods/a1ef6a31-f947-4b17-90ba-5d16957f9ea1/volumes" Nov 04 04:47:42.418863 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.418842 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.418955 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.418871 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.418955 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.418889 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.418955 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.418906 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ktpj\" (UniqueName: \"kubernetes.io/projected/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-kube-api-access-2ktpj\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.419049 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.419024 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-config-out\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.419081 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.419049 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.419081 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.419067 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.419137 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.419080 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.419137 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.419113 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.419209 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.419194 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.419243 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.419236 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.419275 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.419257 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.419359 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.419347 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-config\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.419401 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.419382 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.419443 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.419430 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.419476 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.419450 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.419476 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.419470 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.419575 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.419489 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.419575 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.419562 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.419638 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.419616 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-web-config\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.521857 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.521819 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.522042 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.521868 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.522042 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.521893 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.522042 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.521922 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-config\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.522042 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.521946 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.522042 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.521972 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.522042 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.521996 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.522042 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.522023 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.522042 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.522046 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.522493 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.522074 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.522493 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.522094 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-web-config\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.522493 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.522118 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.522493 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.522146 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.522493 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.522174 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.522493 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.522199 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ktpj\" (UniqueName: \"kubernetes.io/projected/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-kube-api-access-2ktpj\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.522493 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.522246 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-config-out\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.522493 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.522267 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.522493 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.522308 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.522493 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.522368 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.522493 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.522405 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.523036 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.522708 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.523036 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.522766 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.523235 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.523195 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.526125 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.525242 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.526125 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.525265 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.526125 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.525495 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-config\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.526125 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.525560 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.526125 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.525830 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.526125 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.525833 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.526125 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.526076 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-config-out\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.526587 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.526392 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.526587 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.526463 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.526936 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.526892 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.527285 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.527255 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.527809 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.527777 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-web-config\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.528012 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.527994 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.528118 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.528036 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.528747 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.528705 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.529174 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.529008 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.581646 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.581616 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ktpj\" (UniqueName: \"kubernetes.io/projected/9a676d75-8ddf-46b5-8ee1-acdb7f41834d-kube-api-access-2ktpj\") pod \"prometheus-k8s-0\" (UID: \"9a676d75-8ddf-46b5-8ee1-acdb7f41834d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:42.606446 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:42.606419 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:43.027613 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:43.027580 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 04:47:43.030733 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:47:43.030700 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a676d75_8ddf_46b5_8ee1_acdb7f41834d.slice/crio-d971848b31082cb6043adc0efa1c5dea97df0868ab349ee584a35cae9344ea19 WatchSource:0}: Error finding container d971848b31082cb6043adc0efa1c5dea97df0868ab349ee584a35cae9344ea19: Status 404 returned error can't find the container with id d971848b31082cb6043adc0efa1c5dea97df0868ab349ee584a35cae9344ea19 Nov 04 04:47:43.839205 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:43.839165 4395 generic.go:356] "Generic (PLEG): container finished" podID="9a676d75-8ddf-46b5-8ee1-acdb7f41834d" containerID="90855f9cb4fd00663f2ca9695717c9a3a3f3abeb03a3ee6a32edf9ca5308bb1f" exitCode=0 Nov 04 04:47:43.839590 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:43.839211 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9a676d75-8ddf-46b5-8ee1-acdb7f41834d","Type":"ContainerDied","Data":"90855f9cb4fd00663f2ca9695717c9a3a3f3abeb03a3ee6a32edf9ca5308bb1f"} Nov 04 04:47:43.839590 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:43.839280 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9a676d75-8ddf-46b5-8ee1-acdb7f41834d","Type":"ContainerStarted","Data":"d971848b31082cb6043adc0efa1c5dea97df0868ab349ee584a35cae9344ea19"} Nov 04 04:47:44.846084 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:44.846050 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9a676d75-8ddf-46b5-8ee1-acdb7f41834d","Type":"ContainerStarted","Data":"20a0b2a72b5b32aa8d3a3da085aac907cd6ccace8eadbede4e8a6a9ea0e8b288"} Nov 04 04:47:44.846455 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:44.846115 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9a676d75-8ddf-46b5-8ee1-acdb7f41834d","Type":"ContainerStarted","Data":"3126cb3a5302750a8ebd81ab206c195b57aefcc4581bcc21608f04d034108251"} Nov 04 04:47:44.846455 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:44.846141 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9a676d75-8ddf-46b5-8ee1-acdb7f41834d","Type":"ContainerStarted","Data":"60ebe725f8d33bd769eefefd4b687c3050fc35e9431b5a152909f80000cc27e9"} Nov 04 04:47:44.846455 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:44.846165 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9a676d75-8ddf-46b5-8ee1-acdb7f41834d","Type":"ContainerStarted","Data":"af8a1c4fe7b1d04de7e7dba61181f201f052739b2e7d3dcc6654a4c6063556d2"} Nov 04 04:47:44.846455 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:44.846188 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9a676d75-8ddf-46b5-8ee1-acdb7f41834d","Type":"ContainerStarted","Data":"b485eed5d8de0121b4ce46df3904214eb6afb1cc49aceea9f75913f4fc0cad07"} Nov 04 04:47:44.846455 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:44.846210 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9a676d75-8ddf-46b5-8ee1-acdb7f41834d","Type":"ContainerStarted","Data":"63b6ff237fda50441bdbf0f1021c42be1cb0189d2c18c4bcc7c6bd741dba8c18"} Nov 04 04:47:44.971693 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:44.971648 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.955343632 podStartE2EDuration="2.971636696s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.293064ms" isStatefulPod=false podCreationTimestamp="2026-11-04 04:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:47:44.97063819 +0000 UTC m=+106.660720637" watchObservedRunningTime="2026-11-04 04:47:44.971636696 +0000 UTC m=+106.661719142" Nov 04 04:47:47.606992 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:47.606954 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:47:55.042175 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:47:55.042143 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b478c5787-vgmt8"] Nov 04 04:48:03.457833 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:48:03.457786 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 04:48:20.061078 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:48:20.060972 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-b478c5787-vgmt8" podUID="f16979cc-64df-4df4-9498-70d87f82d236" containerName="console" containerID="cri-o://d69cab3ff728d037e078b4611452902009617a4230d8b8c47cf25d0142ce7209" gracePeriod=15 Nov 04 04:48:20.315806 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:48:20.315761 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b478c5787-vgmt8" Nov 04 04:48:20.402766 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:48:20.402740 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f16979cc-64df-4df4-9498-70d87f82d236-console-config\" (UniqueName: \"kubernetes.io/configmap/f16979cc-64df-4df4-9498-70d87f82d236-console-config\") pod \"f16979cc-64df-4df4-9498-70d87f82d236\" (UID: \"f16979cc-64df-4df4-9498-70d87f82d236\") " Nov 04 04:48:20.402892 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:48:20.402771 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f16979cc-64df-4df4-9498-70d87f82d236-service-ca\" (UniqueName: \"kubernetes.io/configmap/f16979cc-64df-4df4-9498-70d87f82d236-service-ca\") pod \"f16979cc-64df-4df4-9498-70d87f82d236\" (UID: \"f16979cc-64df-4df4-9498-70d87f82d236\") " Nov 04 04:48:20.402892 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:48:20.402792 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f16979cc-64df-4df4-9498-70d87f82d236-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f16979cc-64df-4df4-9498-70d87f82d236-trusted-ca-bundle\") pod \"f16979cc-64df-4df4-9498-70d87f82d236\" (UID: \"f16979cc-64df-4df4-9498-70d87f82d236\") " Nov 04 04:48:20.402892 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:48:20.402812 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f16979cc-64df-4df4-9498-70d87f82d236-kube-api-access-t6k8d\" (UniqueName: \"kubernetes.io/projected/f16979cc-64df-4df4-9498-70d87f82d236-kube-api-access-t6k8d\") pod \"f16979cc-64df-4df4-9498-70d87f82d236\" (UID: \"f16979cc-64df-4df4-9498-70d87f82d236\") " Nov 04 04:48:20.402892 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:48:20.402835 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f16979cc-64df-4df4-9498-70d87f82d236-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f16979cc-64df-4df4-9498-70d87f82d236-console-serving-cert\") pod \"f16979cc-64df-4df4-9498-70d87f82d236\" (UID: \"f16979cc-64df-4df4-9498-70d87f82d236\") " Nov 04 04:48:20.402892 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:48:20.402860 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f16979cc-64df-4df4-9498-70d87f82d236-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f16979cc-64df-4df4-9498-70d87f82d236-tmp\") pod \"f16979cc-64df-4df4-9498-70d87f82d236\" (UID: \"f16979cc-64df-4df4-9498-70d87f82d236\") " Nov 04 04:48:20.403163 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:48:20.402926 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f16979cc-64df-4df4-9498-70d87f82d236-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f16979cc-64df-4df4-9498-70d87f82d236-oauth-serving-cert\") pod \"f16979cc-64df-4df4-9498-70d87f82d236\" (UID: \"f16979cc-64df-4df4-9498-70d87f82d236\") " Nov 04 04:48:20.403163 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:48:20.402969 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f16979cc-64df-4df4-9498-70d87f82d236-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f16979cc-64df-4df4-9498-70d87f82d236-console-oauth-config\") pod \"f16979cc-64df-4df4-9498-70d87f82d236\" (UID: \"f16979cc-64df-4df4-9498-70d87f82d236\") " Nov 04 04:48:20.403274 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:48:20.403248 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f16979cc-64df-4df4-9498-70d87f82d236-tmp" pod "f16979cc-64df-4df4-9498-70d87f82d236" (UID: "f16979cc-64df-4df4-9498-70d87f82d236"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:48:20.403362 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:48:20.403187 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f16979cc-64df-4df4-9498-70d87f82d236-service-ca" pod "f16979cc-64df-4df4-9498-70d87f82d236" (UID: "f16979cc-64df-4df4-9498-70d87f82d236"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:48:20.403362 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:48:20.403322 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f16979cc-64df-4df4-9498-70d87f82d236-trusted-ca-bundle" pod "f16979cc-64df-4df4-9498-70d87f82d236" (UID: "f16979cc-64df-4df4-9498-70d87f82d236"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:48:20.403616 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:48:20.403591 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f16979cc-64df-4df4-9498-70d87f82d236-console-config" pod "f16979cc-64df-4df4-9498-70d87f82d236" (UID: "f16979cc-64df-4df4-9498-70d87f82d236"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:48:20.403698 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:48:20.403639 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f16979cc-64df-4df4-9498-70d87f82d236-oauth-serving-cert" pod "f16979cc-64df-4df4-9498-70d87f82d236" (UID: "f16979cc-64df-4df4-9498-70d87f82d236"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 04:48:20.405045 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:48:20.405016 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f16979cc-64df-4df4-9498-70d87f82d236-kube-api-access-t6k8d" pod "f16979cc-64df-4df4-9498-70d87f82d236" (UID: "f16979cc-64df-4df4-9498-70d87f82d236"). InnerVolumeSpecName "kube-api-access-t6k8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:48:20.405143 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:48:20.405064 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f16979cc-64df-4df4-9498-70d87f82d236-console-oauth-config" pod "f16979cc-64df-4df4-9498-70d87f82d236" (UID: "f16979cc-64df-4df4-9498-70d87f82d236"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:48:20.405143 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:48:20.405095 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f16979cc-64df-4df4-9498-70d87f82d236-console-serving-cert" pod "f16979cc-64df-4df4-9498-70d87f82d236" (UID: "f16979cc-64df-4df4-9498-70d87f82d236"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:48:20.503667 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:48:20.503646 4395 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f16979cc-64df-4df4-9498-70d87f82d236-console-oauth-config\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:48:20.503667 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:48:20.503668 4395 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f16979cc-64df-4df4-9498-70d87f82d236-console-config\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:48:20.503825 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:48:20.503677 4395 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f16979cc-64df-4df4-9498-70d87f82d236-service-ca\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:48:20.503825 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:48:20.503686 4395 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f16979cc-64df-4df4-9498-70d87f82d236-trusted-ca-bundle\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:48:20.503825 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:48:20.503695 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6k8d\" (UniqueName: \"kubernetes.io/projected/f16979cc-64df-4df4-9498-70d87f82d236-kube-api-access-t6k8d\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:48:20.503825 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:48:20.503704 4395 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f16979cc-64df-4df4-9498-70d87f82d236-console-serving-cert\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:48:20.503825 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:48:20.503713 4395 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f16979cc-64df-4df4-9498-70d87f82d236-tmp\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:48:20.503825 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:48:20.503721 4395 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f16979cc-64df-4df4-9498-70d87f82d236-oauth-serving-cert\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:48:20.964726 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:48:20.964691 4395 generic.go:356] "Generic (PLEG): container finished" podID="f16979cc-64df-4df4-9498-70d87f82d236" containerID="d69cab3ff728d037e078b4611452902009617a4230d8b8c47cf25d0142ce7209" exitCode=0 Nov 04 04:48:20.964842 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:48:20.964747 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b478c5787-vgmt8" Nov 04 04:48:20.964842 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:48:20.964753 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b478c5787-vgmt8" event={"ID":"f16979cc-64df-4df4-9498-70d87f82d236","Type":"ContainerDied","Data":"d69cab3ff728d037e078b4611452902009617a4230d8b8c47cf25d0142ce7209"} Nov 04 04:48:20.964842 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:48:20.964799 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b478c5787-vgmt8" event={"ID":"f16979cc-64df-4df4-9498-70d87f82d236","Type":"ContainerDied","Data":"474aea2e1e66f95658f8d6a02d1a71c518265d4805e31216219215781ea48b5b"} Nov 04 04:48:20.964842 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:48:20.964821 4395 scope.go:122] "RemoveContainer" containerID="d69cab3ff728d037e078b4611452902009617a4230d8b8c47cf25d0142ce7209" Nov 04 04:48:20.973608 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:48:20.973587 4395 scope.go:122] "RemoveContainer" containerID="d69cab3ff728d037e078b4611452902009617a4230d8b8c47cf25d0142ce7209" Nov 04 04:48:20.973927 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:48:20.973905 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d69cab3ff728d037e078b4611452902009617a4230d8b8c47cf25d0142ce7209\": container with ID starting with d69cab3ff728d037e078b4611452902009617a4230d8b8c47cf25d0142ce7209 not found: ID does not exist" containerID="d69cab3ff728d037e078b4611452902009617a4230d8b8c47cf25d0142ce7209" Nov 04 04:48:20.973978 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:48:20.973936 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d69cab3ff728d037e078b4611452902009617a4230d8b8c47cf25d0142ce7209"} err="failed to get container status \"d69cab3ff728d037e078b4611452902009617a4230d8b8c47cf25d0142ce7209\": rpc error: code = NotFound desc = could not find container \"d69cab3ff728d037e078b4611452902009617a4230d8b8c47cf25d0142ce7209\": container with ID starting with d69cab3ff728d037e078b4611452902009617a4230d8b8c47cf25d0142ce7209 not found: ID does not exist" Nov 04 04:48:21.119117 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:48:21.119090 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b478c5787-vgmt8"] Nov 04 04:48:21.165522 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:48:21.165488 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b478c5787-vgmt8"] Nov 04 04:48:22.413763 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:48:22.413727 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f16979cc-64df-4df4-9498-70d87f82d236" path="/var/lib/kubelet/pods/f16979cc-64df-4df4-9498-70d87f82d236/volumes" Nov 04 04:48:33.456229 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:48:33.456192 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 04:48:42.607054 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:48:42.607013 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:42.623656 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:48:42.623630 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:48:43.059804 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:48:43.059782 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 04:49:03.456610 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:49:03.456573 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 04:49:13.567889 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.567844 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65lpp6"] Nov 04 04:49:13.568572 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.568328 4395 cpu_manager.go:414] "containerMap: removing container" podUID="f16979cc-64df-4df4-9498-70d87f82d236" containerName="console" Nov 04 04:49:13.568572 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.568347 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f16979cc-64df-4df4-9498-70d87f82d236" containerName="console" Nov 04 04:49:13.568572 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.568448 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="f16979cc-64df-4df4-9498-70d87f82d236" containerName="console" Nov 04 04:49:13.571618 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.571592 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65lpp6" Nov 04 04:49:13.657191 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.657161 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d253c2ca-3db9-4066-be2c-b337bf9235b8-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65lpp6\" (UID: \"d253c2ca-3db9-4066-be2c-b337bf9235b8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65lpp6" Nov 04 04:49:13.657335 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.657212 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d253c2ca-3db9-4066-be2c-b337bf9235b8-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65lpp6\" (UID: \"d253c2ca-3db9-4066-be2c-b337bf9235b8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65lpp6" Nov 04 04:49:13.657412 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.657366 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmq2t\" (UniqueName: \"kubernetes.io/projected/d253c2ca-3db9-4066-be2c-b337bf9235b8-kube-api-access-cmq2t\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65lpp6\" (UID: \"d253c2ca-3db9-4066-be2c-b337bf9235b8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65lpp6" Nov 04 04:49:13.667176 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.667157 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 04 04:49:13.667415 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.667398 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-d7kzj\"" Nov 04 04:49:13.667513 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.667434 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 04 04:49:13.668371 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.668354 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65lpp6"] Nov 04 04:49:13.668908 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.668891 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alwkkc"] Nov 04 04:49:13.671929 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.671915 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alwkkc" Nov 04 04:49:13.720984 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.720967 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-lqgxz\"" Nov 04 04:49:13.739949 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.739928 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alwkkc"] Nov 04 04:49:13.758495 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.758474 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmq2t\" (UniqueName: \"kubernetes.io/projected/d253c2ca-3db9-4066-be2c-b337bf9235b8-kube-api-access-cmq2t\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65lpp6\" (UID: \"d253c2ca-3db9-4066-be2c-b337bf9235b8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65lpp6" Nov 04 04:49:13.758584 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.758512 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63b97455-5163-41db-8f71-0b651e1133b5-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alwkkc\" (UID: \"63b97455-5163-41db-8f71-0b651e1133b5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alwkkc" Nov 04 04:49:13.758584 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.758532 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d253c2ca-3db9-4066-be2c-b337bf9235b8-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65lpp6\" (UID: \"d253c2ca-3db9-4066-be2c-b337bf9235b8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65lpp6" Nov 04 04:49:13.758679 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.758659 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63b97455-5163-41db-8f71-0b651e1133b5-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alwkkc\" (UID: \"63b97455-5163-41db-8f71-0b651e1133b5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alwkkc" Nov 04 04:49:13.758742 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.758726 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d253c2ca-3db9-4066-be2c-b337bf9235b8-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65lpp6\" (UID: \"d253c2ca-3db9-4066-be2c-b337bf9235b8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65lpp6" Nov 04 04:49:13.758796 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.758764 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc8nv\" (UniqueName: \"kubernetes.io/projected/63b97455-5163-41db-8f71-0b651e1133b5-kube-api-access-cc8nv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alwkkc\" (UID: \"63b97455-5163-41db-8f71-0b651e1133b5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alwkkc" Nov 04 04:49:13.758852 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.758836 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d253c2ca-3db9-4066-be2c-b337bf9235b8-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65lpp6\" (UID: \"d253c2ca-3db9-4066-be2c-b337bf9235b8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65lpp6" Nov 04 04:49:13.758934 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.758919 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d253c2ca-3db9-4066-be2c-b337bf9235b8-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65lpp6\" (UID: \"d253c2ca-3db9-4066-be2c-b337bf9235b8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65lpp6" Nov 04 04:49:13.766039 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.766020 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7h6nkg"] Nov 04 04:49:13.769268 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.769253 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7h6nkg" Nov 04 04:49:13.816066 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.816039 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmq2t\" (UniqueName: \"kubernetes.io/projected/d253c2ca-3db9-4066-be2c-b337bf9235b8-kube-api-access-cmq2t\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65lpp6\" (UID: \"d253c2ca-3db9-4066-be2c-b337bf9235b8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65lpp6" Nov 04 04:49:13.821155 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.821102 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-shzzs\"" Nov 04 04:49:13.826856 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.826838 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7h6nkg"] Nov 04 04:49:13.859609 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.859585 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63b97455-5163-41db-8f71-0b651e1133b5-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alwkkc\" (UID: \"63b97455-5163-41db-8f71-0b651e1133b5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alwkkc" Nov 04 04:49:13.859686 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.859624 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc8nv\" (UniqueName: \"kubernetes.io/projected/63b97455-5163-41db-8f71-0b651e1133b5-kube-api-access-cc8nv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alwkkc\" (UID: \"63b97455-5163-41db-8f71-0b651e1133b5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alwkkc" Nov 04 04:49:13.859686 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.859649 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04f0a2cd-2384-47f5-817f-0c8c6baf6bfc-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7h6nkg\" (UID: \"04f0a2cd-2384-47f5-817f-0c8c6baf6bfc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7h6nkg" Nov 04 04:49:13.859686 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.859663 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04f0a2cd-2384-47f5-817f-0c8c6baf6bfc-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7h6nkg\" (UID: \"04f0a2cd-2384-47f5-817f-0c8c6baf6bfc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7h6nkg" Nov 04 04:49:13.859790 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.859731 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63b97455-5163-41db-8f71-0b651e1133b5-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alwkkc\" (UID: \"63b97455-5163-41db-8f71-0b651e1133b5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alwkkc" Nov 04 04:49:13.859827 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.859785 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzwzn\" (UniqueName: \"kubernetes.io/projected/04f0a2cd-2384-47f5-817f-0c8c6baf6bfc-kube-api-access-qzwzn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7h6nkg\" (UID: \"04f0a2cd-2384-47f5-817f-0c8c6baf6bfc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7h6nkg" Nov 04 04:49:13.859974 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.859959 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63b97455-5163-41db-8f71-0b651e1133b5-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alwkkc\" (UID: \"63b97455-5163-41db-8f71-0b651e1133b5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alwkkc" Nov 04 04:49:13.860012 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.859979 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63b97455-5163-41db-8f71-0b651e1133b5-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alwkkc\" (UID: \"63b97455-5163-41db-8f71-0b651e1133b5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alwkkc" Nov 04 04:49:13.863029 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.863012 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch468s"] Nov 04 04:49:13.866163 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.866149 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch468s" Nov 04 04:49:13.888086 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.888067 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65lpp6" Nov 04 04:49:13.917094 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.917074 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc8nv\" (UniqueName: \"kubernetes.io/projected/63b97455-5163-41db-8f71-0b651e1133b5-kube-api-access-cc8nv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alwkkc\" (UID: \"63b97455-5163-41db-8f71-0b651e1133b5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alwkkc" Nov 04 04:49:13.921026 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.921003 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-gjnpw\"" Nov 04 04:49:13.925048 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.924986 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch468s"] Nov 04 04:49:13.960398 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.960374 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt9gd\" (UniqueName: \"kubernetes.io/projected/865b3463-eefd-437f-b8bc-b234559d8da3-kube-api-access-vt9gd\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch468s\" (UID: \"865b3463-eefd-437f-b8bc-b234559d8da3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch468s" Nov 04 04:49:13.960538 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.960415 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04f0a2cd-2384-47f5-817f-0c8c6baf6bfc-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7h6nkg\" (UID: \"04f0a2cd-2384-47f5-817f-0c8c6baf6bfc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7h6nkg" Nov 04 04:49:13.960538 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.960434 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04f0a2cd-2384-47f5-817f-0c8c6baf6bfc-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7h6nkg\" (UID: \"04f0a2cd-2384-47f5-817f-0c8c6baf6bfc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7h6nkg" Nov 04 04:49:13.960538 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.960522 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzwzn\" (UniqueName: \"kubernetes.io/projected/04f0a2cd-2384-47f5-817f-0c8c6baf6bfc-kube-api-access-qzwzn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7h6nkg\" (UID: \"04f0a2cd-2384-47f5-817f-0c8c6baf6bfc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7h6nkg" Nov 04 04:49:13.960709 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.960562 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/865b3463-eefd-437f-b8bc-b234559d8da3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch468s\" (UID: \"865b3463-eefd-437f-b8bc-b234559d8da3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch468s" Nov 04 04:49:13.960709 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.960597 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/865b3463-eefd-437f-b8bc-b234559d8da3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch468s\" (UID: \"865b3463-eefd-437f-b8bc-b234559d8da3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch468s" Nov 04 04:49:13.960822 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.960751 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04f0a2cd-2384-47f5-817f-0c8c6baf6bfc-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7h6nkg\" (UID: \"04f0a2cd-2384-47f5-817f-0c8c6baf6bfc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7h6nkg" Nov 04 04:49:13.960925 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.960907 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04f0a2cd-2384-47f5-817f-0c8c6baf6bfc-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7h6nkg\" (UID: \"04f0a2cd-2384-47f5-817f-0c8c6baf6bfc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7h6nkg" Nov 04 04:49:13.986088 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:13.986070 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alwkkc" Nov 04 04:49:14.018062 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:14.018043 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzwzn\" (UniqueName: \"kubernetes.io/projected/04f0a2cd-2384-47f5-817f-0c8c6baf6bfc-kube-api-access-qzwzn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7h6nkg\" (UID: \"04f0a2cd-2384-47f5-817f-0c8c6baf6bfc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7h6nkg" Nov 04 04:49:14.061560 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:14.061535 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/865b3463-eefd-437f-b8bc-b234559d8da3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch468s\" (UID: \"865b3463-eefd-437f-b8bc-b234559d8da3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch468s" Nov 04 04:49:14.061656 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:14.061575 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt9gd\" (UniqueName: \"kubernetes.io/projected/865b3463-eefd-437f-b8bc-b234559d8da3-kube-api-access-vt9gd\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch468s\" (UID: \"865b3463-eefd-437f-b8bc-b234559d8da3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch468s" Nov 04 04:49:14.061656 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:14.061622 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/865b3463-eefd-437f-b8bc-b234559d8da3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch468s\" (UID: \"865b3463-eefd-437f-b8bc-b234559d8da3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch468s" Nov 04 04:49:14.061928 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:14.061908 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/865b3463-eefd-437f-b8bc-b234559d8da3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch468s\" (UID: \"865b3463-eefd-437f-b8bc-b234559d8da3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch468s" Nov 04 04:49:14.061964 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:14.061924 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/865b3463-eefd-437f-b8bc-b234559d8da3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch468s\" (UID: \"865b3463-eefd-437f-b8bc-b234559d8da3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch468s" Nov 04 04:49:14.083975 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:14.083936 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7h6nkg" Nov 04 04:49:14.119561 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:14.119535 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt9gd\" (UniqueName: \"kubernetes.io/projected/865b3463-eefd-437f-b8bc-b234559d8da3-kube-api-access-vt9gd\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch468s\" (UID: \"865b3463-eefd-437f-b8bc-b234559d8da3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch468s" Nov 04 04:49:14.190354 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:14.189945 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch468s" Nov 04 04:49:14.256293 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:14.256272 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65lpp6"] Nov 04 04:49:14.257223 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:49:14.257201 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd253c2ca_3db9_4066_be2c_b337bf9235b8.slice/crio-6bcb2f4bfd5f8964cd1ec910ab302c3feabfa92da73bec833fd780513555ce45 WatchSource:0}: Error finding container 6bcb2f4bfd5f8964cd1ec910ab302c3feabfa92da73bec833fd780513555ce45: Status 404 returned error can't find the container with id 6bcb2f4bfd5f8964cd1ec910ab302c3feabfa92da73bec833fd780513555ce45 Nov 04 04:49:14.349531 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:14.349401 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alwkkc"] Nov 04 04:49:14.351719 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:49:14.351692 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63b97455_5163_41db_8f71_0b651e1133b5.slice/crio-5dadde685ac5999432bf659e0be69a2531f619d8436791ca6e51f6a8a1f510ff WatchSource:0}: Error finding container 5dadde685ac5999432bf659e0be69a2531f619d8436791ca6e51f6a8a1f510ff: Status 404 returned error can't find the container with id 5dadde685ac5999432bf659e0be69a2531f619d8436791ca6e51f6a8a1f510ff Nov 04 04:49:14.443236 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:14.443211 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7h6nkg"] Nov 04 04:49:14.446169 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:49:14.446146 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04f0a2cd_2384_47f5_817f_0c8c6baf6bfc.slice/crio-1d3b4bbeaa049517d48926e09e81b1d4e57fa6828f0d9d19060879f5738b6a82 WatchSource:0}: Error finding container 1d3b4bbeaa049517d48926e09e81b1d4e57fa6828f0d9d19060879f5738b6a82: Status 404 returned error can't find the container with id 1d3b4bbeaa049517d48926e09e81b1d4e57fa6828f0d9d19060879f5738b6a82 Nov 04 04:49:14.552172 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:14.552150 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch468s"] Nov 04 04:49:14.554450 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:49:14.554429 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod865b3463_eefd_437f_b8bc_b234559d8da3.slice/crio-d5c934079431f9a52e1825902b1b5afad1a9dc819a76cbc817886aff91da2a9e WatchSource:0}: Error finding container d5c934079431f9a52e1825902b1b5afad1a9dc819a76cbc817886aff91da2a9e: Status 404 returned error can't find the container with id d5c934079431f9a52e1825902b1b5afad1a9dc819a76cbc817886aff91da2a9e Nov 04 04:49:14.637443 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:14.637396 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsjtq"] Nov 04 04:49:14.643735 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:14.643708 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsjtq" Nov 04 04:49:14.692512 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:14.692490 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-lndcz\"" Nov 04 04:49:14.699081 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:14.699062 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsjtq"] Nov 04 04:49:14.775242 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:14.775222 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqwnc\" (UniqueName: \"kubernetes.io/projected/823f9e8c-7322-4a93-82fc-8e5f7b206b05-kube-api-access-qqwnc\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsjtq\" (UID: \"823f9e8c-7322-4a93-82fc-8e5f7b206b05\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsjtq" Nov 04 04:49:14.775337 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:14.775251 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/823f9e8c-7322-4a93-82fc-8e5f7b206b05-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsjtq\" (UID: \"823f9e8c-7322-4a93-82fc-8e5f7b206b05\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsjtq" Nov 04 04:49:14.775337 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:14.775272 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/823f9e8c-7322-4a93-82fc-8e5f7b206b05-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsjtq\" (UID: \"823f9e8c-7322-4a93-82fc-8e5f7b206b05\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsjtq" Nov 04 04:49:14.875988 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:14.875962 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqwnc\" (UniqueName: \"kubernetes.io/projected/823f9e8c-7322-4a93-82fc-8e5f7b206b05-kube-api-access-qqwnc\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsjtq\" (UID: \"823f9e8c-7322-4a93-82fc-8e5f7b206b05\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsjtq" Nov 04 04:49:14.876077 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:14.875991 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/823f9e8c-7322-4a93-82fc-8e5f7b206b05-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsjtq\" (UID: \"823f9e8c-7322-4a93-82fc-8e5f7b206b05\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsjtq" Nov 04 04:49:14.876077 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:14.876011 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/823f9e8c-7322-4a93-82fc-8e5f7b206b05-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsjtq\" (UID: \"823f9e8c-7322-4a93-82fc-8e5f7b206b05\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsjtq" Nov 04 04:49:14.876309 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:14.876284 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/823f9e8c-7322-4a93-82fc-8e5f7b206b05-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsjtq\" (UID: \"823f9e8c-7322-4a93-82fc-8e5f7b206b05\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsjtq" Nov 04 04:49:14.876351 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:14.876336 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/823f9e8c-7322-4a93-82fc-8e5f7b206b05-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsjtq\" (UID: \"823f9e8c-7322-4a93-82fc-8e5f7b206b05\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsjtq" Nov 04 04:49:14.933151 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:14.933104 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqwnc\" (UniqueName: \"kubernetes.io/projected/823f9e8c-7322-4a93-82fc-8e5f7b206b05-kube-api-access-qqwnc\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsjtq\" (UID: \"823f9e8c-7322-4a93-82fc-8e5f7b206b05\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsjtq" Nov 04 04:49:14.958953 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:14.958932 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsjtq" Nov 04 04:49:15.172598 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:15.172561 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch468s" event={"ID":"865b3463-eefd-437f-b8bc-b234559d8da3","Type":"ContainerStarted","Data":"d5c934079431f9a52e1825902b1b5afad1a9dc819a76cbc817886aff91da2a9e"} Nov 04 04:49:15.173704 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:15.173685 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65lpp6" event={"ID":"d253c2ca-3db9-4066-be2c-b337bf9235b8","Type":"ContainerStarted","Data":"6bcb2f4bfd5f8964cd1ec910ab302c3feabfa92da73bec833fd780513555ce45"} Nov 04 04:49:15.174792 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:15.174766 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7h6nkg" event={"ID":"04f0a2cd-2384-47f5-817f-0c8c6baf6bfc","Type":"ContainerStarted","Data":"1d3b4bbeaa049517d48926e09e81b1d4e57fa6828f0d9d19060879f5738b6a82"} Nov 04 04:49:15.175864 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:15.175842 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alwkkc" event={"ID":"63b97455-5163-41db-8f71-0b651e1133b5","Type":"ContainerStarted","Data":"5dadde685ac5999432bf659e0be69a2531f619d8436791ca6e51f6a8a1f510ff"} Nov 04 04:49:15.331187 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:15.331166 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsjtq"] Nov 04 04:49:15.333678 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:49:15.333656 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod823f9e8c_7322_4a93_82fc_8e5f7b206b05.slice/crio-d3f637a84b3a7748d24c067fae50ee8ccbab0188988507fb3c5b0e6327ca69fd WatchSource:0}: Error finding container d3f637a84b3a7748d24c067fae50ee8ccbab0188988507fb3c5b0e6327ca69fd: Status 404 returned error can't find the container with id d3f637a84b3a7748d24c067fae50ee8ccbab0188988507fb3c5b0e6327ca69fd Nov 04 04:49:16.070896 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:16.070861 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfqk4x"] Nov 04 04:49:16.075394 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:16.075371 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfqk4x" Nov 04 04:49:16.170560 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:16.170531 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-xd7sd\"" Nov 04 04:49:16.171736 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:16.171704 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfqk4x"] Nov 04 04:49:16.182221 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:16.182190 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsjtq" event={"ID":"823f9e8c-7322-4a93-82fc-8e5f7b206b05","Type":"ContainerStarted","Data":"d3f637a84b3a7748d24c067fae50ee8ccbab0188988507fb3c5b0e6327ca69fd"} Nov 04 04:49:16.187082 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:16.186833 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d8af63e-6b0a-4293-b533-0ca5a52101a3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfqk4x\" (UID: \"1d8af63e-6b0a-4293-b533-0ca5a52101a3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfqk4x" Nov 04 04:49:16.187193 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:16.187154 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfwzm\" (UniqueName: \"kubernetes.io/projected/1d8af63e-6b0a-4293-b533-0ca5a52101a3-kube-api-access-lfwzm\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfqk4x\" (UID: \"1d8af63e-6b0a-4293-b533-0ca5a52101a3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfqk4x" Nov 04 04:49:16.187260 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:16.187203 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d8af63e-6b0a-4293-b533-0ca5a52101a3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfqk4x\" (UID: \"1d8af63e-6b0a-4293-b533-0ca5a52101a3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfqk4x" Nov 04 04:49:16.288136 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:16.288097 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d8af63e-6b0a-4293-b533-0ca5a52101a3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfqk4x\" (UID: \"1d8af63e-6b0a-4293-b533-0ca5a52101a3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfqk4x" Nov 04 04:49:16.288280 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:16.288196 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d8af63e-6b0a-4293-b533-0ca5a52101a3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfqk4x\" (UID: \"1d8af63e-6b0a-4293-b533-0ca5a52101a3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfqk4x" Nov 04 04:49:16.288280 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:16.288262 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfwzm\" (UniqueName: \"kubernetes.io/projected/1d8af63e-6b0a-4293-b533-0ca5a52101a3-kube-api-access-lfwzm\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfqk4x\" (UID: \"1d8af63e-6b0a-4293-b533-0ca5a52101a3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfqk4x" Nov 04 04:49:16.288573 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:16.288549 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d8af63e-6b0a-4293-b533-0ca5a52101a3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfqk4x\" (UID: \"1d8af63e-6b0a-4293-b533-0ca5a52101a3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfqk4x" Nov 04 04:49:16.288635 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:16.288574 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d8af63e-6b0a-4293-b533-0ca5a52101a3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfqk4x\" (UID: \"1d8af63e-6b0a-4293-b533-0ca5a52101a3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfqk4x" Nov 04 04:49:16.346897 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:16.346840 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfwzm\" (UniqueName: \"kubernetes.io/projected/1d8af63e-6b0a-4293-b533-0ca5a52101a3-kube-api-access-lfwzm\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfqk4x\" (UID: \"1d8af63e-6b0a-4293-b533-0ca5a52101a3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfqk4x" Nov 04 04:49:16.395655 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:16.395619 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfqk4x" Nov 04 04:49:16.782728 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:16.782695 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfqk4x"] Nov 04 04:49:17.191454 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:17.191380 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfqk4x" event={"ID":"1d8af63e-6b0a-4293-b533-0ca5a52101a3","Type":"ContainerStarted","Data":"fe577777221b217b4fe83c1b48f14ce94d362f253df6069ec73800f363c36e36"} Nov 04 04:49:19.042673 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.042632 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tp4xv"] Nov 04 04:49:19.046185 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.046162 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tp4xv" Nov 04 04:49:19.140987 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.140952 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-87jsw\"" Nov 04 04:49:19.142234 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.142169 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tp4xv"] Nov 04 04:49:19.216003 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.215974 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05387354-389e-4173-85a5-5562506e71e4-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tp4xv\" (UID: \"05387354-389e-4173-85a5-5562506e71e4\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tp4xv" Nov 04 04:49:19.216190 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.216023 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05387354-389e-4173-85a5-5562506e71e4-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tp4xv\" (UID: \"05387354-389e-4173-85a5-5562506e71e4\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tp4xv" Nov 04 04:49:19.216190 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.216154 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv2q9\" (UniqueName: \"kubernetes.io/projected/05387354-389e-4173-85a5-5562506e71e4-kube-api-access-cv2q9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tp4xv\" (UID: \"05387354-389e-4173-85a5-5562506e71e4\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tp4xv" Nov 04 04:49:19.317435 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.317396 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv2q9\" (UniqueName: \"kubernetes.io/projected/05387354-389e-4173-85a5-5562506e71e4-kube-api-access-cv2q9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tp4xv\" (UID: \"05387354-389e-4173-85a5-5562506e71e4\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tp4xv" Nov 04 04:49:19.317610 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.317487 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05387354-389e-4173-85a5-5562506e71e4-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tp4xv\" (UID: \"05387354-389e-4173-85a5-5562506e71e4\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tp4xv" Nov 04 04:49:19.317610 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.317514 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05387354-389e-4173-85a5-5562506e71e4-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tp4xv\" (UID: \"05387354-389e-4173-85a5-5562506e71e4\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tp4xv" Nov 04 04:49:19.317976 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.317954 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05387354-389e-4173-85a5-5562506e71e4-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tp4xv\" (UID: \"05387354-389e-4173-85a5-5562506e71e4\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tp4xv" Nov 04 04:49:19.318572 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.318505 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05387354-389e-4173-85a5-5562506e71e4-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tp4xv\" (UID: \"05387354-389e-4173-85a5-5562506e71e4\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tp4xv" Nov 04 04:49:19.375729 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.375703 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv2q9\" (UniqueName: \"kubernetes.io/projected/05387354-389e-4173-85a5-5562506e71e4-kube-api-access-cv2q9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tp4xv\" (UID: \"05387354-389e-4173-85a5-5562506e71e4\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tp4xv" Nov 04 04:49:19.439495 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.439469 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl"] Nov 04 04:49:19.445394 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.445373 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" Nov 04 04:49:19.494373 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.494348 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-7dlcv\"" Nov 04 04:49:19.494373 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.494371 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 04 04:49:19.494557 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.494430 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 04 04:49:19.494557 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.494453 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 04 04:49:19.505161 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.505138 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl"] Nov 04 04:49:19.619854 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.619783 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115eb9f0-c97e-4133-939f-1797a4157820-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qrtfl\" (UID: \"115eb9f0-c97e-4133-939f-1797a4157820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" Nov 04 04:49:19.619854 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.619818 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115eb9f0-c97e-4133-939f-1797a4157820-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qrtfl\" (UID: \"115eb9f0-c97e-4133-939f-1797a4157820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" Nov 04 04:49:19.619854 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.619835 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115eb9f0-c97e-4133-939f-1797a4157820-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qrtfl\" (UID: \"115eb9f0-c97e-4133-939f-1797a4157820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" Nov 04 04:49:19.619854 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.619851 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115eb9f0-c97e-4133-939f-1797a4157820-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qrtfl\" (UID: \"115eb9f0-c97e-4133-939f-1797a4157820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" Nov 04 04:49:19.620164 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.619955 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115eb9f0-c97e-4133-939f-1797a4157820-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qrtfl\" (UID: \"115eb9f0-c97e-4133-939f-1797a4157820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" Nov 04 04:49:19.620164 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.620017 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115eb9f0-c97e-4133-939f-1797a4157820-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qrtfl\" (UID: \"115eb9f0-c97e-4133-939f-1797a4157820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" Nov 04 04:49:19.620275 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.620166 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115eb9f0-c97e-4133-939f-1797a4157820-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qrtfl\" (UID: \"115eb9f0-c97e-4133-939f-1797a4157820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" Nov 04 04:49:19.620275 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.620215 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzzw8\" (UniqueName: \"kubernetes.io/projected/115eb9f0-c97e-4133-939f-1797a4157820-kube-api-access-hzzw8\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qrtfl\" (UID: \"115eb9f0-c97e-4133-939f-1797a4157820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" Nov 04 04:49:19.620461 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.620325 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/115eb9f0-c97e-4133-939f-1797a4157820-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qrtfl\" (UID: \"115eb9f0-c97e-4133-939f-1797a4157820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" Nov 04 04:49:19.667876 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.667849 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tp4xv" Nov 04 04:49:19.720953 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.720922 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115eb9f0-c97e-4133-939f-1797a4157820-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qrtfl\" (UID: \"115eb9f0-c97e-4133-939f-1797a4157820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" Nov 04 04:49:19.721107 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.720981 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzzw8\" (UniqueName: \"kubernetes.io/projected/115eb9f0-c97e-4133-939f-1797a4157820-kube-api-access-hzzw8\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qrtfl\" (UID: \"115eb9f0-c97e-4133-939f-1797a4157820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" Nov 04 04:49:19.721107 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.721018 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/115eb9f0-c97e-4133-939f-1797a4157820-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qrtfl\" (UID: \"115eb9f0-c97e-4133-939f-1797a4157820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" Nov 04 04:49:19.721107 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.721044 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115eb9f0-c97e-4133-939f-1797a4157820-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qrtfl\" (UID: \"115eb9f0-c97e-4133-939f-1797a4157820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" Nov 04 04:49:19.721107 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.721067 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115eb9f0-c97e-4133-939f-1797a4157820-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qrtfl\" (UID: \"115eb9f0-c97e-4133-939f-1797a4157820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" Nov 04 04:49:19.721107 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.721083 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115eb9f0-c97e-4133-939f-1797a4157820-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qrtfl\" (UID: \"115eb9f0-c97e-4133-939f-1797a4157820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" Nov 04 04:49:19.721107 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.721105 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115eb9f0-c97e-4133-939f-1797a4157820-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qrtfl\" (UID: \"115eb9f0-c97e-4133-939f-1797a4157820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" Nov 04 04:49:19.721462 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.721123 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115eb9f0-c97e-4133-939f-1797a4157820-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qrtfl\" (UID: \"115eb9f0-c97e-4133-939f-1797a4157820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" Nov 04 04:49:19.721462 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.721138 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115eb9f0-c97e-4133-939f-1797a4157820-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qrtfl\" (UID: \"115eb9f0-c97e-4133-939f-1797a4157820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" Nov 04 04:49:19.721633 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.721610 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115eb9f0-c97e-4133-939f-1797a4157820-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qrtfl\" (UID: \"115eb9f0-c97e-4133-939f-1797a4157820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" Nov 04 04:49:19.721803 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.721783 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115eb9f0-c97e-4133-939f-1797a4157820-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qrtfl\" (UID: \"115eb9f0-c97e-4133-939f-1797a4157820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" Nov 04 04:49:19.722008 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.721986 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115eb9f0-c97e-4133-939f-1797a4157820-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qrtfl\" (UID: \"115eb9f0-c97e-4133-939f-1797a4157820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" Nov 04 04:49:19.722085 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.722052 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115eb9f0-c97e-4133-939f-1797a4157820-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qrtfl\" (UID: \"115eb9f0-c97e-4133-939f-1797a4157820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" Nov 04 04:49:19.722448 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.722410 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115eb9f0-c97e-4133-939f-1797a4157820-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qrtfl\" (UID: \"115eb9f0-c97e-4133-939f-1797a4157820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" Nov 04 04:49:19.725750 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.725669 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115eb9f0-c97e-4133-939f-1797a4157820-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qrtfl\" (UID: \"115eb9f0-c97e-4133-939f-1797a4157820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" Nov 04 04:49:19.737374 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.737291 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115eb9f0-c97e-4133-939f-1797a4157820-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qrtfl\" (UID: \"115eb9f0-c97e-4133-939f-1797a4157820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" Nov 04 04:49:19.778992 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.778966 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/115eb9f0-c97e-4133-939f-1797a4157820-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qrtfl\" (UID: \"115eb9f0-c97e-4133-939f-1797a4157820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" Nov 04 04:49:19.779602 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:19.779581 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzzw8\" (UniqueName: \"kubernetes.io/projected/115eb9f0-c97e-4133-939f-1797a4157820-kube-api-access-hzzw8\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qrtfl\" (UID: \"115eb9f0-c97e-4133-939f-1797a4157820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" Nov 04 04:49:20.056315 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:20.056265 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tp4xv"] Nov 04 04:49:20.059691 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:20.059666 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" Nov 04 04:49:20.060023 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:49:20.059995 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05387354_389e_4173_85a5_5562506e71e4.slice/crio-d5e10b82a4ca6873814fb651bd067109678db507db482bb2500dd049b9c40940 WatchSource:0}: Error finding container d5e10b82a4ca6873814fb651bd067109678db507db482bb2500dd049b9c40940: Status 404 returned error can't find the container with id d5e10b82a4ca6873814fb651bd067109678db507db482bb2500dd049b9c40940 Nov 04 04:49:20.210076 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:20.210038 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tp4xv" event={"ID":"05387354-389e-4173-85a5-5562506e71e4","Type":"ContainerStarted","Data":"d5e10b82a4ca6873814fb651bd067109678db507db482bb2500dd049b9c40940"} Nov 04 04:49:20.499682 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:20.499608 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl"] Nov 04 04:49:20.507930 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:49:20.507899 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod115eb9f0_c97e_4133_939f_1797a4157820.slice/crio-757c5c56566d7b55b19d145c38f06bd79aeb8d99fc8074103805a7171691e25b WatchSource:0}: Error finding container 757c5c56566d7b55b19d145c38f06bd79aeb8d99fc8074103805a7171691e25b: Status 404 returned error can't find the container with id 757c5c56566d7b55b19d145c38f06bd79aeb8d99fc8074103805a7171691e25b Nov 04 04:49:21.220702 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:21.220666 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" event={"ID":"115eb9f0-c97e-4133-939f-1797a4157820","Type":"ContainerStarted","Data":"757c5c56566d7b55b19d145c38f06bd79aeb8d99fc8074103805a7171691e25b"} Nov 04 04:49:27.259655 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:27.259573 4395 generic.go:356] "Generic (PLEG): container finished" podID="865b3463-eefd-437f-b8bc-b234559d8da3" containerID="f1315440ab674a966139ce68aff161c09ae2989f57afd7a6d3d68e43d74b5654" exitCode=0 Nov 04 04:49:27.260080 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:27.259688 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch468s" event={"ID":"865b3463-eefd-437f-b8bc-b234559d8da3","Type":"ContainerDied","Data":"f1315440ab674a966139ce68aff161c09ae2989f57afd7a6d3d68e43d74b5654"} Nov 04 04:49:27.265250 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:27.265225 4395 generic.go:356] "Generic (PLEG): container finished" podID="63b97455-5163-41db-8f71-0b651e1133b5" containerID="9e785fb3b28a72efae00056f065ee4a94181ac1cea7d3a761b5067ad93c1d197" exitCode=0 Nov 04 04:49:27.265373 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:27.265289 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alwkkc" event={"ID":"63b97455-5163-41db-8f71-0b651e1133b5","Type":"ContainerDied","Data":"9e785fb3b28a72efae00056f065ee4a94181ac1cea7d3a761b5067ad93c1d197"} Nov 04 04:49:27.875953 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:49:27.875927 4395 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d8af63e_6b0a_4293_b533_0ca5a52101a3.slice/crio-7f524aec60ed19735782c376413ae0df4f38e28d933cb1c86bc4a1ac41d3f6d1.scope\": RecentStats: unable to find data in memory cache]" Nov 04 04:49:28.202682 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:28.202654 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 04:49:28.202755 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:28.202722 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 04:49:28.202755 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:28.202750 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 04:49:28.270139 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:28.270115 4395 generic.go:356] "Generic (PLEG): container finished" podID="1d8af63e-6b0a-4293-b533-0ca5a52101a3" containerID="7f524aec60ed19735782c376413ae0df4f38e28d933cb1c86bc4a1ac41d3f6d1" exitCode=0 Nov 04 04:49:28.270456 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:28.270185 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfqk4x" event={"ID":"1d8af63e-6b0a-4293-b533-0ca5a52101a3","Type":"ContainerDied","Data":"7f524aec60ed19735782c376413ae0df4f38e28d933cb1c86bc4a1ac41d3f6d1"} Nov 04 04:49:28.271806 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:28.271782 4395 generic.go:356] "Generic (PLEG): container finished" podID="04f0a2cd-2384-47f5-817f-0c8c6baf6bfc" containerID="25a68f1037a5ddfa249da7d1971cdff129446390db3d30f188d23fef1463c710" exitCode=0 Nov 04 04:49:28.271906 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:28.271840 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7h6nkg" event={"ID":"04f0a2cd-2384-47f5-817f-0c8c6baf6bfc","Type":"ContainerDied","Data":"25a68f1037a5ddfa249da7d1971cdff129446390db3d30f188d23fef1463c710"} Nov 04 04:49:29.276512 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:29.276423 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" event={"ID":"115eb9f0-c97e-4133-939f-1797a4157820","Type":"ContainerStarted","Data":"5346ce7606aa23646860e1c24f3939bf7231b3d84c1c22054764b003337b0440"} Nov 04 04:49:29.277839 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:29.277815 4395 generic.go:356] "Generic (PLEG): container finished" podID="823f9e8c-7322-4a93-82fc-8e5f7b206b05" containerID="e27701c17fd78bfda1251ec85a7d15cf4406c3eb32a424f45b4a94d75cd746e6" exitCode=0 Nov 04 04:49:29.277942 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:29.277886 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsjtq" event={"ID":"823f9e8c-7322-4a93-82fc-8e5f7b206b05","Type":"ContainerDied","Data":"e27701c17fd78bfda1251ec85a7d15cf4406c3eb32a424f45b4a94d75cd746e6"} Nov 04 04:49:29.279416 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:29.279394 4395 generic.go:356] "Generic (PLEG): container finished" podID="d253c2ca-3db9-4066-be2c-b337bf9235b8" containerID="cd286a4572d3f00fd7e200bac55e83ae8fca5b274339704893337f709f249087" exitCode=0 Nov 04 04:49:29.279532 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:29.279458 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65lpp6" event={"ID":"d253c2ca-3db9-4066-be2c-b337bf9235b8","Type":"ContainerDied","Data":"cd286a4572d3f00fd7e200bac55e83ae8fca5b274339704893337f709f249087"} Nov 04 04:49:29.280917 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:29.280891 4395 generic.go:356] "Generic (PLEG): container finished" podID="05387354-389e-4173-85a5-5562506e71e4" containerID="23d166d363ea464bb8b5c48828e0f1dd93167dc7b530c14ca284e688113f3ee3" exitCode=0 Nov 04 04:49:29.281023 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:29.280928 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tp4xv" event={"ID":"05387354-389e-4173-85a5-5562506e71e4","Type":"ContainerDied","Data":"23d166d363ea464bb8b5c48828e0f1dd93167dc7b530c14ca284e688113f3ee3"} Nov 04 04:49:29.389444 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:29.389403 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" podStartSLOduration=2.702985487 podStartE2EDuration="10.38939007s" totalImagesPullingTime="7.686404583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:49:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:49:29.388221255 +0000 UTC m=+211.078303696" watchObservedRunningTime="2026-11-04 04:49:29.38939007 +0000 UTC m=+211.079472513" Nov 04 04:49:30.059884 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:30.059854 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" Nov 04 04:49:30.064507 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:30.064487 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" Nov 04 04:49:30.286778 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:30.286739 4395 generic.go:356] "Generic (PLEG): container finished" podID="865b3463-eefd-437f-b8bc-b234559d8da3" containerID="884e4c24bfc5839f7251b91226579b437fe548237041febec35299cca537d609" exitCode=0 Nov 04 04:49:30.287217 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:30.286845 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch468s" event={"ID":"865b3463-eefd-437f-b8bc-b234559d8da3","Type":"ContainerDied","Data":"884e4c24bfc5839f7251b91226579b437fe548237041febec35299cca537d609"} Nov 04 04:49:30.289493 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:30.289464 4395 generic.go:356] "Generic (PLEG): container finished" podID="63b97455-5163-41db-8f71-0b651e1133b5" containerID="0aa1d8d047006251a9220bc7a611dfa46d19577b7c9b6ea9fb8d1c7a8344cf8f" exitCode=0 Nov 04 04:49:30.289618 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:30.289547 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alwkkc" event={"ID":"63b97455-5163-41db-8f71-0b651e1133b5","Type":"ContainerDied","Data":"0aa1d8d047006251a9220bc7a611dfa46d19577b7c9b6ea9fb8d1c7a8344cf8f"} Nov 04 04:49:30.289859 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:30.289839 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" Nov 04 04:49:30.290977 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:30.290958 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" Nov 04 04:49:31.296751 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:31.296691 4395 generic.go:356] "Generic (PLEG): container finished" podID="04f0a2cd-2384-47f5-817f-0c8c6baf6bfc" containerID="fd35160fd98b314dc9cb450a21d799d3253e0eee8a232778569532e4bfcba844" exitCode=0 Nov 04 04:49:31.297169 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:31.297025 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7h6nkg" event={"ID":"04f0a2cd-2384-47f5-817f-0c8c6baf6bfc","Type":"ContainerDied","Data":"fd35160fd98b314dc9cb450a21d799d3253e0eee8a232778569532e4bfcba844"} Nov 04 04:49:31.299289 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:31.299262 4395 generic.go:356] "Generic (PLEG): container finished" podID="05387354-389e-4173-85a5-5562506e71e4" containerID="f4c9ce1701fc5a8fb996a6dfb1f2f93b1a631acdb2cc401a26c6fc8f5b5447af" exitCode=0 Nov 04 04:49:31.299435 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:31.299374 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tp4xv" event={"ID":"05387354-389e-4173-85a5-5562506e71e4","Type":"ContainerDied","Data":"f4c9ce1701fc5a8fb996a6dfb1f2f93b1a631acdb2cc401a26c6fc8f5b5447af"} Nov 04 04:49:32.306017 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:32.305985 4395 generic.go:356] "Generic (PLEG): container finished" podID="1d8af63e-6b0a-4293-b533-0ca5a52101a3" containerID="23dc4e8e873864d032076b9a8dfe56c46524aa3756160e384bb3f284ad96736d" exitCode=0 Nov 04 04:49:32.306504 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:32.306024 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfqk4x" event={"ID":"1d8af63e-6b0a-4293-b533-0ca5a52101a3","Type":"ContainerDied","Data":"23dc4e8e873864d032076b9a8dfe56c46524aa3756160e384bb3f284ad96736d"} Nov 04 04:49:32.308473 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:32.308450 4395 generic.go:356] "Generic (PLEG): container finished" podID="d253c2ca-3db9-4066-be2c-b337bf9235b8" containerID="ae3aca6e622546f2ee8121fc44593f04d9c774a5a96788d6efafc4cce7523979" exitCode=0 Nov 04 04:49:32.309637 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:32.309606 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65lpp6" event={"ID":"d253c2ca-3db9-4066-be2c-b337bf9235b8","Type":"ContainerDied","Data":"ae3aca6e622546f2ee8121fc44593f04d9c774a5a96788d6efafc4cce7523979"} Nov 04 04:49:33.468206 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:49:33.468163 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 04:49:35.336210 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:35.336054 4395 generic.go:356] "Generic (PLEG): container finished" podID="823f9e8c-7322-4a93-82fc-8e5f7b206b05" containerID="59a6754590e821449243876ecc3ef8dbd04ada856898d609a2003d5a4fb93210" exitCode=0 Nov 04 04:49:35.336210 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:35.336117 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsjtq" event={"ID":"823f9e8c-7322-4a93-82fc-8e5f7b206b05","Type":"ContainerDied","Data":"59a6754590e821449243876ecc3ef8dbd04ada856898d609a2003d5a4fb93210"} Nov 04 04:49:45.405738 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:45.405707 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alwkkc" event={"ID":"63b97455-5163-41db-8f71-0b651e1133b5","Type":"ContainerStarted","Data":"13839ceab30a93b30c4eab60e5528af3ed78cf027ecd23ddda1bbd393e48c691"} Nov 04 04:49:46.418578 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:46.418487 4395 generic.go:356] "Generic (PLEG): container finished" podID="865b3463-eefd-437f-b8bc-b234559d8da3" containerID="24b05af02dd51109ec96ca748b29fce7ed19f2f9925312a02ac0a4ebf5e7861a" exitCode=0 Nov 04 04:49:46.418986 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:46.418607 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch468s" event={"ID":"865b3463-eefd-437f-b8bc-b234559d8da3","Type":"ContainerDied","Data":"24b05af02dd51109ec96ca748b29fce7ed19f2f9925312a02ac0a4ebf5e7861a"} Nov 04 04:49:46.421379 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:46.421352 4395 generic.go:356] "Generic (PLEG): container finished" podID="d253c2ca-3db9-4066-be2c-b337bf9235b8" containerID="35c09598ff069a411048475a8e37a2625fdaf504358f48bd2ee5d08ef0aca55c" exitCode=0 Nov 04 04:49:46.421500 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:46.421429 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65lpp6" event={"ID":"d253c2ca-3db9-4066-be2c-b337bf9235b8","Type":"ContainerDied","Data":"35c09598ff069a411048475a8e37a2625fdaf504358f48bd2ee5d08ef0aca55c"} Nov 04 04:49:46.424933 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:46.424912 4395 generic.go:356] "Generic (PLEG): container finished" podID="05387354-389e-4173-85a5-5562506e71e4" containerID="e8d2d8bd9e9606164dd66ab686116917b2c34b501d5adaf49526ce743194c9eb" exitCode=0 Nov 04 04:49:46.425088 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:46.425072 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tp4xv" event={"ID":"05387354-389e-4173-85a5-5562506e71e4","Type":"ContainerDied","Data":"e8d2d8bd9e9606164dd66ab686116917b2c34b501d5adaf49526ce743194c9eb"} Nov 04 04:49:46.428195 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:46.428170 4395 generic.go:356] "Generic (PLEG): container finished" podID="63b97455-5163-41db-8f71-0b651e1133b5" containerID="13839ceab30a93b30c4eab60e5528af3ed78cf027ecd23ddda1bbd393e48c691" exitCode=0 Nov 04 04:49:46.428294 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:46.428212 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alwkkc" event={"ID":"63b97455-5163-41db-8f71-0b651e1133b5","Type":"ContainerDied","Data":"13839ceab30a93b30c4eab60e5528af3ed78cf027ecd23ddda1bbd393e48c691"} Nov 04 04:49:46.607598 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:46.607567 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alwkkc" Nov 04 04:49:46.666416 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:46.666386 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63b97455-5163-41db-8f71-0b651e1133b5-kube-api-access-cc8nv\" (UniqueName: \"kubernetes.io/projected/63b97455-5163-41db-8f71-0b651e1133b5-kube-api-access-cc8nv\") pod \"63b97455-5163-41db-8f71-0b651e1133b5\" (UID: \"63b97455-5163-41db-8f71-0b651e1133b5\") " Nov 04 04:49:46.666568 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:46.666441 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63b97455-5163-41db-8f71-0b651e1133b5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/63b97455-5163-41db-8f71-0b651e1133b5-bundle\") pod \"63b97455-5163-41db-8f71-0b651e1133b5\" (UID: \"63b97455-5163-41db-8f71-0b651e1133b5\") " Nov 04 04:49:46.666634 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:46.666580 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63b97455-5163-41db-8f71-0b651e1133b5-util\" (UniqueName: \"kubernetes.io/empty-dir/63b97455-5163-41db-8f71-0b651e1133b5-util\") pod \"63b97455-5163-41db-8f71-0b651e1133b5\" (UID: \"63b97455-5163-41db-8f71-0b651e1133b5\") " Nov 04 04:49:46.667286 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:46.667252 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63b97455-5163-41db-8f71-0b651e1133b5-bundle" pod "63b97455-5163-41db-8f71-0b651e1133b5" (UID: "63b97455-5163-41db-8f71-0b651e1133b5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:49:46.669005 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:46.668948 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63b97455-5163-41db-8f71-0b651e1133b5-kube-api-access-cc8nv" pod "63b97455-5163-41db-8f71-0b651e1133b5" (UID: "63b97455-5163-41db-8f71-0b651e1133b5"). InnerVolumeSpecName "kube-api-access-cc8nv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:49:46.671640 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:46.671612 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63b97455-5163-41db-8f71-0b651e1133b5-util" pod "63b97455-5163-41db-8f71-0b651e1133b5" (UID: "63b97455-5163-41db-8f71-0b651e1133b5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:49:46.767823 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:46.767793 4395 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63b97455-5163-41db-8f71-0b651e1133b5-util\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:49:46.767976 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:46.767828 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc8nv\" (UniqueName: \"kubernetes.io/projected/63b97455-5163-41db-8f71-0b651e1133b5-kube-api-access-cc8nv\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:49:46.767976 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:46.767843 4395 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63b97455-5163-41db-8f71-0b651e1133b5-bundle\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:49:47.435283 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:47.434840 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7h6nkg" event={"ID":"04f0a2cd-2384-47f5-817f-0c8c6baf6bfc","Type":"ContainerStarted","Data":"dd2f79b8bb3c3a2dea596f9712eaa673135b764dcfae57097aea5d9b1098bfd5"} Nov 04 04:49:47.457328 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:47.455168 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alwkkc" Nov 04 04:49:47.457650 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:47.457625 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alwkkc" event={"ID":"63b97455-5163-41db-8f71-0b651e1133b5","Type":"ContainerDied","Data":"5dadde685ac5999432bf659e0be69a2531f619d8436791ca6e51f6a8a1f510ff"} Nov 04 04:49:47.457692 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:47.457672 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5dadde685ac5999432bf659e0be69a2531f619d8436791ca6e51f6a8a1f510ff" Nov 04 04:49:47.549992 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:47.549329 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7h6nkg" podStartSLOduration=3.322599618 podStartE2EDuration="34.549309606s" totalImagesPullingTime="31.226709988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:49:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:49:47.547772694 +0000 UTC m=+229.237855142" watchObservedRunningTime="2026-11-04 04:49:47.549309606 +0000 UTC m=+229.239392046" Nov 04 04:49:47.639787 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:47.639763 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tp4xv" Nov 04 04:49:47.669706 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:47.669680 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65lpp6" Nov 04 04:49:47.675011 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:47.674982 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05387354-389e-4173-85a5-5562506e71e4-bundle\" (UniqueName: \"kubernetes.io/empty-dir/05387354-389e-4173-85a5-5562506e71e4-bundle\") pod \"05387354-389e-4173-85a5-5562506e71e4\" (UID: \"05387354-389e-4173-85a5-5562506e71e4\") " Nov 04 04:49:47.675125 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:47.675025 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05387354-389e-4173-85a5-5562506e71e4-kube-api-access-cv2q9\" (UniqueName: \"kubernetes.io/projected/05387354-389e-4173-85a5-5562506e71e4-kube-api-access-cv2q9\") pod \"05387354-389e-4173-85a5-5562506e71e4\" (UID: \"05387354-389e-4173-85a5-5562506e71e4\") " Nov 04 04:49:47.675125 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:47.675106 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05387354-389e-4173-85a5-5562506e71e4-util\" (UniqueName: \"kubernetes.io/empty-dir/05387354-389e-4173-85a5-5562506e71e4-util\") pod \"05387354-389e-4173-85a5-5562506e71e4\" (UID: \"05387354-389e-4173-85a5-5562506e71e4\") " Nov 04 04:49:47.675593 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:47.675563 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05387354-389e-4173-85a5-5562506e71e4-bundle" pod "05387354-389e-4173-85a5-5562506e71e4" (UID: "05387354-389e-4173-85a5-5562506e71e4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:49:47.677146 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:47.677123 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05387354-389e-4173-85a5-5562506e71e4-kube-api-access-cv2q9" pod "05387354-389e-4173-85a5-5562506e71e4" (UID: "05387354-389e-4173-85a5-5562506e71e4"). InnerVolumeSpecName "kube-api-access-cv2q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:49:47.681754 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:47.681700 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05387354-389e-4173-85a5-5562506e71e4-util" pod "05387354-389e-4173-85a5-5562506e71e4" (UID: "05387354-389e-4173-85a5-5562506e71e4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:49:47.681856 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:47.681836 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch468s" Nov 04 04:49:47.775572 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:47.775544 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d253c2ca-3db9-4066-be2c-b337bf9235b8-util\" (UniqueName: \"kubernetes.io/empty-dir/d253c2ca-3db9-4066-be2c-b337bf9235b8-util\") pod \"d253c2ca-3db9-4066-be2c-b337bf9235b8\" (UID: \"d253c2ca-3db9-4066-be2c-b337bf9235b8\") " Nov 04 04:49:47.775707 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:47.775582 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/865b3463-eefd-437f-b8bc-b234559d8da3-kube-api-access-vt9gd\" (UniqueName: \"kubernetes.io/projected/865b3463-eefd-437f-b8bc-b234559d8da3-kube-api-access-vt9gd\") pod \"865b3463-eefd-437f-b8bc-b234559d8da3\" (UID: \"865b3463-eefd-437f-b8bc-b234559d8da3\") " Nov 04 04:49:47.775707 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:47.775629 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d253c2ca-3db9-4066-be2c-b337bf9235b8-kube-api-access-cmq2t\" (UniqueName: \"kubernetes.io/projected/d253c2ca-3db9-4066-be2c-b337bf9235b8-kube-api-access-cmq2t\") pod \"d253c2ca-3db9-4066-be2c-b337bf9235b8\" (UID: \"d253c2ca-3db9-4066-be2c-b337bf9235b8\") " Nov 04 04:49:47.775707 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:47.775682 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/865b3463-eefd-437f-b8bc-b234559d8da3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/865b3463-eefd-437f-b8bc-b234559d8da3-bundle\") pod \"865b3463-eefd-437f-b8bc-b234559d8da3\" (UID: \"865b3463-eefd-437f-b8bc-b234559d8da3\") " Nov 04 04:49:47.775882 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:47.775735 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/865b3463-eefd-437f-b8bc-b234559d8da3-util\" (UniqueName: \"kubernetes.io/empty-dir/865b3463-eefd-437f-b8bc-b234559d8da3-util\") pod \"865b3463-eefd-437f-b8bc-b234559d8da3\" (UID: \"865b3463-eefd-437f-b8bc-b234559d8da3\") " Nov 04 04:49:47.775882 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:47.775779 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d253c2ca-3db9-4066-be2c-b337bf9235b8-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d253c2ca-3db9-4066-be2c-b337bf9235b8-bundle\") pod \"d253c2ca-3db9-4066-be2c-b337bf9235b8\" (UID: \"d253c2ca-3db9-4066-be2c-b337bf9235b8\") " Nov 04 04:49:47.776012 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:47.775994 4395 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05387354-389e-4173-85a5-5562506e71e4-util\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:49:47.776071 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:47.776017 4395 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05387354-389e-4173-85a5-5562506e71e4-bundle\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:49:47.776071 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:47.776032 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv2q9\" (UniqueName: \"kubernetes.io/projected/05387354-389e-4173-85a5-5562506e71e4-kube-api-access-cv2q9\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:49:47.776784 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:47.776477 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865b3463-eefd-437f-b8bc-b234559d8da3-bundle" pod "865b3463-eefd-437f-b8bc-b234559d8da3" (UID: "865b3463-eefd-437f-b8bc-b234559d8da3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:49:47.776784 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:47.776756 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d253c2ca-3db9-4066-be2c-b337bf9235b8-bundle" pod "d253c2ca-3db9-4066-be2c-b337bf9235b8" (UID: "d253c2ca-3db9-4066-be2c-b337bf9235b8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:49:47.779714 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:47.779680 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d253c2ca-3db9-4066-be2c-b337bf9235b8-kube-api-access-cmq2t" pod "d253c2ca-3db9-4066-be2c-b337bf9235b8" (UID: "d253c2ca-3db9-4066-be2c-b337bf9235b8"). InnerVolumeSpecName "kube-api-access-cmq2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:49:47.782017 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:47.781993 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d253c2ca-3db9-4066-be2c-b337bf9235b8-util" pod "d253c2ca-3db9-4066-be2c-b337bf9235b8" (UID: "d253c2ca-3db9-4066-be2c-b337bf9235b8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:49:47.782110 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:47.782087 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865b3463-eefd-437f-b8bc-b234559d8da3-util" pod "865b3463-eefd-437f-b8bc-b234559d8da3" (UID: "865b3463-eefd-437f-b8bc-b234559d8da3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:49:47.783679 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:47.783656 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/865b3463-eefd-437f-b8bc-b234559d8da3-kube-api-access-vt9gd" pod "865b3463-eefd-437f-b8bc-b234559d8da3" (UID: "865b3463-eefd-437f-b8bc-b234559d8da3"). InnerVolumeSpecName "kube-api-access-vt9gd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:49:47.877091 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:47.877068 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmq2t\" (UniqueName: \"kubernetes.io/projected/d253c2ca-3db9-4066-be2c-b337bf9235b8-kube-api-access-cmq2t\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:49:47.877188 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:47.877093 4395 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/865b3463-eefd-437f-b8bc-b234559d8da3-bundle\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:49:47.877188 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:47.877107 4395 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/865b3463-eefd-437f-b8bc-b234559d8da3-util\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:49:47.877188 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:47.877121 4395 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d253c2ca-3db9-4066-be2c-b337bf9235b8-bundle\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:49:47.877188 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:47.877133 4395 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d253c2ca-3db9-4066-be2c-b337bf9235b8-util\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:49:47.877188 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:47.877146 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt9gd\" (UniqueName: \"kubernetes.io/projected/865b3463-eefd-437f-b8bc-b234559d8da3-kube-api-access-vt9gd\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:49:48.466482 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:48.466440 4395 generic.go:356] "Generic (PLEG): container finished" podID="04f0a2cd-2384-47f5-817f-0c8c6baf6bfc" containerID="dd2f79b8bb3c3a2dea596f9712eaa673135b764dcfae57097aea5d9b1098bfd5" exitCode=0 Nov 04 04:49:48.466859 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:48.466541 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7h6nkg" event={"ID":"04f0a2cd-2384-47f5-817f-0c8c6baf6bfc","Type":"ContainerDied","Data":"dd2f79b8bb3c3a2dea596f9712eaa673135b764dcfae57097aea5d9b1098bfd5"} Nov 04 04:49:48.468544 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:48.468524 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch468s" event={"ID":"865b3463-eefd-437f-b8bc-b234559d8da3","Type":"ContainerDied","Data":"d5c934079431f9a52e1825902b1b5afad1a9dc819a76cbc817886aff91da2a9e"} Nov 04 04:49:48.468544 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:48.468536 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch468s" Nov 04 04:49:48.468685 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:48.468554 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5c934079431f9a52e1825902b1b5afad1a9dc819a76cbc817886aff91da2a9e" Nov 04 04:49:48.470738 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:48.470715 4395 generic.go:356] "Generic (PLEG): container finished" podID="823f9e8c-7322-4a93-82fc-8e5f7b206b05" containerID="d6525b77810d869e319d5b23ad134785d3722d8b10fe8680efd738e0a548f123" exitCode=0 Nov 04 04:49:48.470837 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:48.470785 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsjtq" event={"ID":"823f9e8c-7322-4a93-82fc-8e5f7b206b05","Type":"ContainerDied","Data":"d6525b77810d869e319d5b23ad134785d3722d8b10fe8680efd738e0a548f123"} Nov 04 04:49:48.472633 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:48.472615 4395 generic.go:356] "Generic (PLEG): container finished" podID="1d8af63e-6b0a-4293-b533-0ca5a52101a3" containerID="8b96769de3daf5c13a1554cb8ad197aaab7613bdc38d1382696978497336fb05" exitCode=0 Nov 04 04:49:48.472706 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:48.472674 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfqk4x" event={"ID":"1d8af63e-6b0a-4293-b533-0ca5a52101a3","Type":"ContainerDied","Data":"8b96769de3daf5c13a1554cb8ad197aaab7613bdc38d1382696978497336fb05"} Nov 04 04:49:48.474210 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:48.474193 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65lpp6" event={"ID":"d253c2ca-3db9-4066-be2c-b337bf9235b8","Type":"ContainerDied","Data":"6bcb2f4bfd5f8964cd1ec910ab302c3feabfa92da73bec833fd780513555ce45"} Nov 04 04:49:48.474276 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:48.474210 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65lpp6" Nov 04 04:49:48.474276 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:48.474221 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bcb2f4bfd5f8964cd1ec910ab302c3feabfa92da73bec833fd780513555ce45" Nov 04 04:49:48.476181 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:48.476159 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tp4xv" event={"ID":"05387354-389e-4173-85a5-5562506e71e4","Type":"ContainerDied","Data":"d5e10b82a4ca6873814fb651bd067109678db507db482bb2500dd049b9c40940"} Nov 04 04:49:48.476276 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:48.476188 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tp4xv" Nov 04 04:49:48.476276 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:48.476195 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5e10b82a4ca6873814fb651bd067109678db507db482bb2500dd049b9c40940" Nov 04 04:49:49.624912 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:49.624888 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsjtq" Nov 04 04:49:49.626379 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:49.626364 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7h6nkg" Nov 04 04:49:49.651445 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:49.651428 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfqk4x" Nov 04 04:49:49.689926 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:49.689902 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823f9e8c-7322-4a93-82fc-8e5f7b206b05-util\" (UniqueName: \"kubernetes.io/empty-dir/823f9e8c-7322-4a93-82fc-8e5f7b206b05-util\") pod \"823f9e8c-7322-4a93-82fc-8e5f7b206b05\" (UID: \"823f9e8c-7322-4a93-82fc-8e5f7b206b05\") " Nov 04 04:49:49.690029 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:49.689945 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04f0a2cd-2384-47f5-817f-0c8c6baf6bfc-bundle\" (UniqueName: \"kubernetes.io/empty-dir/04f0a2cd-2384-47f5-817f-0c8c6baf6bfc-bundle\") pod \"04f0a2cd-2384-47f5-817f-0c8c6baf6bfc\" (UID: \"04f0a2cd-2384-47f5-817f-0c8c6baf6bfc\") " Nov 04 04:49:49.690029 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:49.689976 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d8af63e-6b0a-4293-b533-0ca5a52101a3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d8af63e-6b0a-4293-b533-0ca5a52101a3-bundle\") pod \"1d8af63e-6b0a-4293-b533-0ca5a52101a3\" (UID: \"1d8af63e-6b0a-4293-b533-0ca5a52101a3\") " Nov 04 04:49:49.690029 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:49.690013 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04f0a2cd-2384-47f5-817f-0c8c6baf6bfc-kube-api-access-qzwzn\" (UniqueName: \"kubernetes.io/projected/04f0a2cd-2384-47f5-817f-0c8c6baf6bfc-kube-api-access-qzwzn\") pod \"04f0a2cd-2384-47f5-817f-0c8c6baf6bfc\" (UID: \"04f0a2cd-2384-47f5-817f-0c8c6baf6bfc\") " Nov 04 04:49:49.690183 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:49.690070 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d8af63e-6b0a-4293-b533-0ca5a52101a3-kube-api-access-lfwzm\" (UniqueName: \"kubernetes.io/projected/1d8af63e-6b0a-4293-b533-0ca5a52101a3-kube-api-access-lfwzm\") pod \"1d8af63e-6b0a-4293-b533-0ca5a52101a3\" (UID: \"1d8af63e-6b0a-4293-b533-0ca5a52101a3\") " Nov 04 04:49:49.690183 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:49.690121 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d8af63e-6b0a-4293-b533-0ca5a52101a3-util\" (UniqueName: \"kubernetes.io/empty-dir/1d8af63e-6b0a-4293-b533-0ca5a52101a3-util\") pod \"1d8af63e-6b0a-4293-b533-0ca5a52101a3\" (UID: \"1d8af63e-6b0a-4293-b533-0ca5a52101a3\") " Nov 04 04:49:49.690183 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:49.690152 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/823f9e8c-7322-4a93-82fc-8e5f7b206b05-kube-api-access-qqwnc\" (UniqueName: \"kubernetes.io/projected/823f9e8c-7322-4a93-82fc-8e5f7b206b05-kube-api-access-qqwnc\") pod \"823f9e8c-7322-4a93-82fc-8e5f7b206b05\" (UID: \"823f9e8c-7322-4a93-82fc-8e5f7b206b05\") " Nov 04 04:49:49.690339 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:49.690192 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04f0a2cd-2384-47f5-817f-0c8c6baf6bfc-util\" (UniqueName: \"kubernetes.io/empty-dir/04f0a2cd-2384-47f5-817f-0c8c6baf6bfc-util\") pod \"04f0a2cd-2384-47f5-817f-0c8c6baf6bfc\" (UID: \"04f0a2cd-2384-47f5-817f-0c8c6baf6bfc\") " Nov 04 04:49:49.690339 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:49.690223 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823f9e8c-7322-4a93-82fc-8e5f7b206b05-bundle\" (UniqueName: \"kubernetes.io/empty-dir/823f9e8c-7322-4a93-82fc-8e5f7b206b05-bundle\") pod \"823f9e8c-7322-4a93-82fc-8e5f7b206b05\" (UID: \"823f9e8c-7322-4a93-82fc-8e5f7b206b05\") " Nov 04 04:49:49.690745 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:49.690488 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04f0a2cd-2384-47f5-817f-0c8c6baf6bfc-bundle" pod "04f0a2cd-2384-47f5-817f-0c8c6baf6bfc" (UID: "04f0a2cd-2384-47f5-817f-0c8c6baf6bfc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:49:49.691102 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:49.691075 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d8af63e-6b0a-4293-b533-0ca5a52101a3-bundle" pod "1d8af63e-6b0a-4293-b533-0ca5a52101a3" (UID: "1d8af63e-6b0a-4293-b533-0ca5a52101a3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:49:49.691558 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:49.691534 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823f9e8c-7322-4a93-82fc-8e5f7b206b05-bundle" pod "823f9e8c-7322-4a93-82fc-8e5f7b206b05" (UID: "823f9e8c-7322-4a93-82fc-8e5f7b206b05"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:49:49.692578 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:49.692546 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04f0a2cd-2384-47f5-817f-0c8c6baf6bfc-kube-api-access-qzwzn" pod "04f0a2cd-2384-47f5-817f-0c8c6baf6bfc" (UID: "04f0a2cd-2384-47f5-817f-0c8c6baf6bfc"). InnerVolumeSpecName "kube-api-access-qzwzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:49:49.692678 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:49.692654 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d8af63e-6b0a-4293-b533-0ca5a52101a3-kube-api-access-lfwzm" pod "1d8af63e-6b0a-4293-b533-0ca5a52101a3" (UID: "1d8af63e-6b0a-4293-b533-0ca5a52101a3"). InnerVolumeSpecName "kube-api-access-lfwzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:49:49.692884 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:49.692850 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/823f9e8c-7322-4a93-82fc-8e5f7b206b05-kube-api-access-qqwnc" pod "823f9e8c-7322-4a93-82fc-8e5f7b206b05" (UID: "823f9e8c-7322-4a93-82fc-8e5f7b206b05"). InnerVolumeSpecName "kube-api-access-qqwnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:49:49.697010 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:49.696986 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d8af63e-6b0a-4293-b533-0ca5a52101a3-util" pod "1d8af63e-6b0a-4293-b533-0ca5a52101a3" (UID: "1d8af63e-6b0a-4293-b533-0ca5a52101a3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:49:49.697880 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:49.697862 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04f0a2cd-2384-47f5-817f-0c8c6baf6bfc-util" pod "04f0a2cd-2384-47f5-817f-0c8c6baf6bfc" (UID: "04f0a2cd-2384-47f5-817f-0c8c6baf6bfc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:49:49.698229 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:49.698209 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823f9e8c-7322-4a93-82fc-8e5f7b206b05-util" pod "823f9e8c-7322-4a93-82fc-8e5f7b206b05" (UID: "823f9e8c-7322-4a93-82fc-8e5f7b206b05"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 04:49:49.791130 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:49.791079 4395 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/823f9e8c-7322-4a93-82fc-8e5f7b206b05-util\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:49:49.791130 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:49.791098 4395 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04f0a2cd-2384-47f5-817f-0c8c6baf6bfc-bundle\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:49:49.791130 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:49.791107 4395 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d8af63e-6b0a-4293-b533-0ca5a52101a3-bundle\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:49:49.791130 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:49.791116 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzwzn\" (UniqueName: \"kubernetes.io/projected/04f0a2cd-2384-47f5-817f-0c8c6baf6bfc-kube-api-access-qzwzn\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:49:49.791130 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:49.791125 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfwzm\" (UniqueName: \"kubernetes.io/projected/1d8af63e-6b0a-4293-b533-0ca5a52101a3-kube-api-access-lfwzm\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:49:49.791130 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:49.791134 4395 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d8af63e-6b0a-4293-b533-0ca5a52101a3-util\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:49:49.791383 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:49.791142 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqwnc\" (UniqueName: \"kubernetes.io/projected/823f9e8c-7322-4a93-82fc-8e5f7b206b05-kube-api-access-qqwnc\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:49:49.791383 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:49.791150 4395 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04f0a2cd-2384-47f5-817f-0c8c6baf6bfc-util\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:49:49.791383 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:49.791159 4395 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/823f9e8c-7322-4a93-82fc-8e5f7b206b05-bundle\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:49:50.485765 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:50.485741 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7h6nkg" Nov 04 04:49:50.485916 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:50.485769 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7h6nkg" event={"ID":"04f0a2cd-2384-47f5-817f-0c8c6baf6bfc","Type":"ContainerDied","Data":"1d3b4bbeaa049517d48926e09e81b1d4e57fa6828f0d9d19060879f5738b6a82"} Nov 04 04:49:50.485916 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:50.485817 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d3b4bbeaa049517d48926e09e81b1d4e57fa6828f0d9d19060879f5738b6a82" Nov 04 04:49:50.487770 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:50.487752 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsjtq" Nov 04 04:49:50.487878 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:50.487750 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsjtq" event={"ID":"823f9e8c-7322-4a93-82fc-8e5f7b206b05","Type":"ContainerDied","Data":"d3f637a84b3a7748d24c067fae50ee8ccbab0188988507fb3c5b0e6327ca69fd"} Nov 04 04:49:50.487878 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:50.487862 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3f637a84b3a7748d24c067fae50ee8ccbab0188988507fb3c5b0e6327ca69fd" Nov 04 04:49:50.489802 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:50.489779 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfqk4x" event={"ID":"1d8af63e-6b0a-4293-b533-0ca5a52101a3","Type":"ContainerDied","Data":"fe577777221b217b4fe83c1b48f14ce94d362f253df6069ec73800f363c36e36"} Nov 04 04:49:50.489881 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:50.489821 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe577777221b217b4fe83c1b48f14ce94d362f253df6069ec73800f363c36e36" Nov 04 04:49:50.489881 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:50.489790 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfqk4x" Nov 04 04:49:51.856281 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856240 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-546b6bbf6-ltkh7"] Nov 04 04:49:51.856750 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856667 4395 cpu_manager.go:414] "containerMap: removing container" podUID="04f0a2cd-2384-47f5-817f-0c8c6baf6bfc" containerName="extract" Nov 04 04:49:51.856750 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856685 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04f0a2cd-2384-47f5-817f-0c8c6baf6bfc" containerName="extract" Nov 04 04:49:51.856750 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856699 4395 cpu_manager.go:414] "containerMap: removing container" podUID="823f9e8c-7322-4a93-82fc-8e5f7b206b05" containerName="pull" Nov 04 04:49:51.856750 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856704 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="823f9e8c-7322-4a93-82fc-8e5f7b206b05" containerName="pull" Nov 04 04:49:51.856750 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856713 4395 cpu_manager.go:414] "containerMap: removing container" podUID="d253c2ca-3db9-4066-be2c-b337bf9235b8" containerName="pull" Nov 04 04:49:51.856750 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856718 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d253c2ca-3db9-4066-be2c-b337bf9235b8" containerName="pull" Nov 04 04:49:51.856750 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856728 4395 cpu_manager.go:414] "containerMap: removing container" podUID="04f0a2cd-2384-47f5-817f-0c8c6baf6bfc" containerName="pull" Nov 04 04:49:51.856750 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856733 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04f0a2cd-2384-47f5-817f-0c8c6baf6bfc" containerName="pull" Nov 04 04:49:51.856750 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856740 4395 cpu_manager.go:414] "containerMap: removing container" podUID="865b3463-eefd-437f-b8bc-b234559d8da3" containerName="extract" Nov 04 04:49:51.856750 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856745 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="865b3463-eefd-437f-b8bc-b234559d8da3" containerName="extract" Nov 04 04:49:51.856750 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856751 4395 cpu_manager.go:414] "containerMap: removing container" podUID="1d8af63e-6b0a-4293-b533-0ca5a52101a3" containerName="util" Nov 04 04:49:51.856750 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856756 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d8af63e-6b0a-4293-b533-0ca5a52101a3" containerName="util" Nov 04 04:49:51.857157 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856764 4395 cpu_manager.go:414] "containerMap: removing container" podUID="1d8af63e-6b0a-4293-b533-0ca5a52101a3" containerName="pull" Nov 04 04:49:51.857157 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856769 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d8af63e-6b0a-4293-b533-0ca5a52101a3" containerName="pull" Nov 04 04:49:51.857157 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856776 4395 cpu_manager.go:414] "containerMap: removing container" podUID="d253c2ca-3db9-4066-be2c-b337bf9235b8" containerName="util" Nov 04 04:49:51.857157 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856780 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d253c2ca-3db9-4066-be2c-b337bf9235b8" containerName="util" Nov 04 04:49:51.857157 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856792 4395 cpu_manager.go:414] "containerMap: removing container" podUID="865b3463-eefd-437f-b8bc-b234559d8da3" containerName="pull" Nov 04 04:49:51.857157 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856797 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="865b3463-eefd-437f-b8bc-b234559d8da3" containerName="pull" Nov 04 04:49:51.857157 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856803 4395 cpu_manager.go:414] "containerMap: removing container" podUID="d253c2ca-3db9-4066-be2c-b337bf9235b8" containerName="extract" Nov 04 04:49:51.857157 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856808 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d253c2ca-3db9-4066-be2c-b337bf9235b8" containerName="extract" Nov 04 04:49:51.857157 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856815 4395 cpu_manager.go:414] "containerMap: removing container" podUID="823f9e8c-7322-4a93-82fc-8e5f7b206b05" containerName="extract" Nov 04 04:49:51.857157 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856819 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="823f9e8c-7322-4a93-82fc-8e5f7b206b05" containerName="extract" Nov 04 04:49:51.857157 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856828 4395 cpu_manager.go:414] "containerMap: removing container" podUID="865b3463-eefd-437f-b8bc-b234559d8da3" containerName="util" Nov 04 04:49:51.857157 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856832 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="865b3463-eefd-437f-b8bc-b234559d8da3" containerName="util" Nov 04 04:49:51.857157 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856837 4395 cpu_manager.go:414] "containerMap: removing container" podUID="823f9e8c-7322-4a93-82fc-8e5f7b206b05" containerName="util" Nov 04 04:49:51.857157 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856842 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="823f9e8c-7322-4a93-82fc-8e5f7b206b05" containerName="util" Nov 04 04:49:51.857157 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856847 4395 cpu_manager.go:414] "containerMap: removing container" podUID="63b97455-5163-41db-8f71-0b651e1133b5" containerName="extract" Nov 04 04:49:51.857157 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856852 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63b97455-5163-41db-8f71-0b651e1133b5" containerName="extract" Nov 04 04:49:51.857157 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856860 4395 cpu_manager.go:414] "containerMap: removing container" podUID="63b97455-5163-41db-8f71-0b651e1133b5" containerName="pull" Nov 04 04:49:51.857157 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856865 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63b97455-5163-41db-8f71-0b651e1133b5" containerName="pull" Nov 04 04:49:51.857157 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856871 4395 cpu_manager.go:414] "containerMap: removing container" podUID="05387354-389e-4173-85a5-5562506e71e4" containerName="util" Nov 04 04:49:51.857157 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856875 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05387354-389e-4173-85a5-5562506e71e4" containerName="util" Nov 04 04:49:51.857157 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856882 4395 cpu_manager.go:414] "containerMap: removing container" podUID="1d8af63e-6b0a-4293-b533-0ca5a52101a3" containerName="extract" Nov 04 04:49:51.857157 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856886 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d8af63e-6b0a-4293-b533-0ca5a52101a3" containerName="extract" Nov 04 04:49:51.857157 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856895 4395 cpu_manager.go:414] "containerMap: removing container" podUID="04f0a2cd-2384-47f5-817f-0c8c6baf6bfc" containerName="util" Nov 04 04:49:51.857157 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856899 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04f0a2cd-2384-47f5-817f-0c8c6baf6bfc" containerName="util" Nov 04 04:49:51.857157 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856907 4395 cpu_manager.go:414] "containerMap: removing container" podUID="05387354-389e-4173-85a5-5562506e71e4" containerName="pull" Nov 04 04:49:51.857157 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856911 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05387354-389e-4173-85a5-5562506e71e4" containerName="pull" Nov 04 04:49:51.857157 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856917 4395 cpu_manager.go:414] "containerMap: removing container" podUID="63b97455-5163-41db-8f71-0b651e1133b5" containerName="util" Nov 04 04:49:51.857157 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856922 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63b97455-5163-41db-8f71-0b651e1133b5" containerName="util" Nov 04 04:49:51.857157 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856928 4395 cpu_manager.go:414] "containerMap: removing container" podUID="05387354-389e-4173-85a5-5562506e71e4" containerName="extract" Nov 04 04:49:51.857157 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856932 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05387354-389e-4173-85a5-5562506e71e4" containerName="extract" Nov 04 04:49:51.857157 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856972 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="05387354-389e-4173-85a5-5562506e71e4" containerName="extract" Nov 04 04:49:51.857157 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856978 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="04f0a2cd-2384-47f5-817f-0c8c6baf6bfc" containerName="extract" Nov 04 04:49:51.857157 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856985 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="865b3463-eefd-437f-b8bc-b234559d8da3" containerName="extract" Nov 04 04:49:51.857157 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856993 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="d253c2ca-3db9-4066-be2c-b337bf9235b8" containerName="extract" Nov 04 04:49:51.857157 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.856999 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="823f9e8c-7322-4a93-82fc-8e5f7b206b05" containerName="extract" Nov 04 04:49:51.857157 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.857005 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="63b97455-5163-41db-8f71-0b651e1133b5" containerName="extract" Nov 04 04:49:51.857157 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.857011 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d8af63e-6b0a-4293-b533-0ca5a52101a3" containerName="extract" Nov 04 04:49:51.859817 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.859800 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-546b6bbf6-ltkh7" Nov 04 04:49:51.907902 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.907877 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9sdv\" (UniqueName: \"kubernetes.io/projected/d7540c40-08d3-4c7c-a2b7-be2b32c75986-kube-api-access-d9sdv\") pod \"opendatahub-operator-controller-manager-546b6bbf6-ltkh7\" (UID: \"d7540c40-08d3-4c7c-a2b7-be2b32c75986\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-546b6bbf6-ltkh7" Nov 04 04:49:51.908017 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.907914 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d7540c40-08d3-4c7c-a2b7-be2b32c75986-webhook-cert\") pod \"opendatahub-operator-controller-manager-546b6bbf6-ltkh7\" (UID: \"d7540c40-08d3-4c7c-a2b7-be2b32c75986\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-546b6bbf6-ltkh7" Nov 04 04:49:51.908017 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.907933 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d7540c40-08d3-4c7c-a2b7-be2b32c75986-metrics-cert\") pod \"opendatahub-operator-controller-manager-546b6bbf6-ltkh7\" (UID: \"d7540c40-08d3-4c7c-a2b7-be2b32c75986\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-546b6bbf6-ltkh7" Nov 04 04:49:51.908094 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.908021 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d7540c40-08d3-4c7c-a2b7-be2b32c75986-manifests\") pod \"opendatahub-operator-controller-manager-546b6bbf6-ltkh7\" (UID: \"d7540c40-08d3-4c7c-a2b7-be2b32c75986\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-546b6bbf6-ltkh7" Nov 04 04:49:51.908136 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.908089 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d7540c40-08d3-4c7c-a2b7-be2b32c75986-apiservice-cert\") pod \"opendatahub-operator-controller-manager-546b6bbf6-ltkh7\" (UID: \"d7540c40-08d3-4c7c-a2b7-be2b32c75986\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-546b6bbf6-ltkh7" Nov 04 04:49:51.908136 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.908127 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7540c40-08d3-4c7c-a2b7-be2b32c75986-tmp\") pod \"opendatahub-operator-controller-manager-546b6bbf6-ltkh7\" (UID: \"d7540c40-08d3-4c7c-a2b7-be2b32c75986\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-546b6bbf6-ltkh7" Nov 04 04:49:51.947343 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.947323 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 04 04:49:51.948924 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.948908 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 04 04:49:51.966984 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.966959 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 04 04:49:51.978983 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.978954 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 04 04:49:51.979060 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.979003 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-jhscp\"" Nov 04 04:49:51.979100 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:51.979076 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 04 04:49:52.009210 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:52.009188 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d7540c40-08d3-4c7c-a2b7-be2b32c75986-webhook-cert\") pod \"opendatahub-operator-controller-manager-546b6bbf6-ltkh7\" (UID: \"d7540c40-08d3-4c7c-a2b7-be2b32c75986\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-546b6bbf6-ltkh7" Nov 04 04:49:52.009279 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:52.009213 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d7540c40-08d3-4c7c-a2b7-be2b32c75986-metrics-cert\") pod \"opendatahub-operator-controller-manager-546b6bbf6-ltkh7\" (UID: \"d7540c40-08d3-4c7c-a2b7-be2b32c75986\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-546b6bbf6-ltkh7" Nov 04 04:49:52.009279 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:52.009239 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d7540c40-08d3-4c7c-a2b7-be2b32c75986-manifests\") pod \"opendatahub-operator-controller-manager-546b6bbf6-ltkh7\" (UID: \"d7540c40-08d3-4c7c-a2b7-be2b32c75986\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-546b6bbf6-ltkh7" Nov 04 04:49:52.009279 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:52.009268 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d7540c40-08d3-4c7c-a2b7-be2b32c75986-apiservice-cert\") pod \"opendatahub-operator-controller-manager-546b6bbf6-ltkh7\" (UID: \"d7540c40-08d3-4c7c-a2b7-be2b32c75986\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-546b6bbf6-ltkh7" Nov 04 04:49:52.009434 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:52.009294 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7540c40-08d3-4c7c-a2b7-be2b32c75986-tmp\") pod \"opendatahub-operator-controller-manager-546b6bbf6-ltkh7\" (UID: \"d7540c40-08d3-4c7c-a2b7-be2b32c75986\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-546b6bbf6-ltkh7" Nov 04 04:49:52.009434 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:52.009375 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9sdv\" (UniqueName: \"kubernetes.io/projected/d7540c40-08d3-4c7c-a2b7-be2b32c75986-kube-api-access-d9sdv\") pod \"opendatahub-operator-controller-manager-546b6bbf6-ltkh7\" (UID: \"d7540c40-08d3-4c7c-a2b7-be2b32c75986\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-546b6bbf6-ltkh7" Nov 04 04:49:52.009626 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:52.009610 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d7540c40-08d3-4c7c-a2b7-be2b32c75986-manifests\") pod \"opendatahub-operator-controller-manager-546b6bbf6-ltkh7\" (UID: \"d7540c40-08d3-4c7c-a2b7-be2b32c75986\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-546b6bbf6-ltkh7" Nov 04 04:49:52.011691 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:52.011668 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7540c40-08d3-4c7c-a2b7-be2b32c75986-tmp\") pod \"opendatahub-operator-controller-manager-546b6bbf6-ltkh7\" (UID: \"d7540c40-08d3-4c7c-a2b7-be2b32c75986\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-546b6bbf6-ltkh7" Nov 04 04:49:52.011781 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:52.011678 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d7540c40-08d3-4c7c-a2b7-be2b32c75986-apiservice-cert\") pod \"opendatahub-operator-controller-manager-546b6bbf6-ltkh7\" (UID: \"d7540c40-08d3-4c7c-a2b7-be2b32c75986\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-546b6bbf6-ltkh7" Nov 04 04:49:52.011907 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:52.011893 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d7540c40-08d3-4c7c-a2b7-be2b32c75986-metrics-cert\") pod \"opendatahub-operator-controller-manager-546b6bbf6-ltkh7\" (UID: \"d7540c40-08d3-4c7c-a2b7-be2b32c75986\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-546b6bbf6-ltkh7" Nov 04 04:49:52.012025 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:52.012005 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d7540c40-08d3-4c7c-a2b7-be2b32c75986-webhook-cert\") pod \"opendatahub-operator-controller-manager-546b6bbf6-ltkh7\" (UID: \"d7540c40-08d3-4c7c-a2b7-be2b32c75986\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-546b6bbf6-ltkh7" Nov 04 04:49:52.016071 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:52.016046 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-546b6bbf6-ltkh7"] Nov 04 04:49:52.070907 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:52.070885 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9sdv\" (UniqueName: \"kubernetes.io/projected/d7540c40-08d3-4c7c-a2b7-be2b32c75986-kube-api-access-d9sdv\") pod \"opendatahub-operator-controller-manager-546b6bbf6-ltkh7\" (UID: \"d7540c40-08d3-4c7c-a2b7-be2b32c75986\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-546b6bbf6-ltkh7" Nov 04 04:49:52.173769 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:52.173722 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-546b6bbf6-ltkh7" Nov 04 04:49:52.677833 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:52.677738 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-546b6bbf6-ltkh7"] Nov 04 04:49:52.680336 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:49:52.680291 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7540c40_08d3_4c7c_a2b7_be2b32c75986.slice/crio-10588638fefe9ba4807cea5ea45895966c5e3d0fdccdf2debb612a88a486e7a3 WatchSource:0}: Error finding container 10588638fefe9ba4807cea5ea45895966c5e3d0fdccdf2debb612a88a486e7a3: Status 404 returned error can't find the container with id 10588638fefe9ba4807cea5ea45895966c5e3d0fdccdf2debb612a88a486e7a3 Nov 04 04:49:53.504458 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:53.504425 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-546b6bbf6-ltkh7" event={"ID":"d7540c40-08d3-4c7c-a2b7-be2b32c75986","Type":"ContainerStarted","Data":"10588638fefe9ba4807cea5ea45895966c5e3d0fdccdf2debb612a88a486e7a3"} Nov 04 04:49:57.027607 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:57.027566 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-x4jsm"] Nov 04 04:49:57.030073 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:57.030050 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-x4jsm" Nov 04 04:49:57.081760 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:57.081733 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 04 04:49:57.128014 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:57.127990 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 04 04:49:57.129408 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:57.129385 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-x4jsm"] Nov 04 04:49:57.150323 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:57.147318 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nl5p\" (UniqueName: \"kubernetes.io/projected/43063223-b5c0-4618-8d01-6a8dde7bc38c-kube-api-access-7nl5p\") pod \"jobset-operator-674cd59dc5-x4jsm\" (UID: \"43063223-b5c0-4618-8d01-6a8dde7bc38c\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-x4jsm" Nov 04 04:49:57.150323 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:57.147453 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43063223-b5c0-4618-8d01-6a8dde7bc38c-tmp\") pod \"jobset-operator-674cd59dc5-x4jsm\" (UID: \"43063223-b5c0-4618-8d01-6a8dde7bc38c\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-x4jsm" Nov 04 04:49:57.174554 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:57.174531 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-5v7jh\"" Nov 04 04:49:57.248465 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:57.248434 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nl5p\" (UniqueName: \"kubernetes.io/projected/43063223-b5c0-4618-8d01-6a8dde7bc38c-kube-api-access-7nl5p\") pod \"jobset-operator-674cd59dc5-x4jsm\" (UID: \"43063223-b5c0-4618-8d01-6a8dde7bc38c\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-x4jsm" Nov 04 04:49:57.248565 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:57.248501 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43063223-b5c0-4618-8d01-6a8dde7bc38c-tmp\") pod \"jobset-operator-674cd59dc5-x4jsm\" (UID: \"43063223-b5c0-4618-8d01-6a8dde7bc38c\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-x4jsm" Nov 04 04:49:57.248897 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:57.248877 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43063223-b5c0-4618-8d01-6a8dde7bc38c-tmp\") pod \"jobset-operator-674cd59dc5-x4jsm\" (UID: \"43063223-b5c0-4618-8d01-6a8dde7bc38c\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-x4jsm" Nov 04 04:49:57.334233 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:57.334210 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nl5p\" (UniqueName: \"kubernetes.io/projected/43063223-b5c0-4618-8d01-6a8dde7bc38c-kube-api-access-7nl5p\") pod \"jobset-operator-674cd59dc5-x4jsm\" (UID: \"43063223-b5c0-4618-8d01-6a8dde7bc38c\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-x4jsm" Nov 04 04:49:57.352917 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:57.352893 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-x4jsm" Nov 04 04:49:57.733908 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:57.733888 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-x4jsm"] Nov 04 04:49:57.736051 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:49:57.736026 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43063223_b5c0_4618_8d01_6a8dde7bc38c.slice/crio-f9992801156602fa9dc976009ff56f905b40951a9c937eb2a86c6e60e3db82b3 WatchSource:0}: Error finding container f9992801156602fa9dc976009ff56f905b40951a9c937eb2a86c6e60e3db82b3: Status 404 returned error can't find the container with id f9992801156602fa9dc976009ff56f905b40951a9c937eb2a86c6e60e3db82b3 Nov 04 04:49:58.530264 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:58.530188 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-x4jsm" event={"ID":"43063223-b5c0-4618-8d01-6a8dde7bc38c","Type":"ContainerStarted","Data":"f9992801156602fa9dc976009ff56f905b40951a9c937eb2a86c6e60e3db82b3"} Nov 04 04:49:58.531728 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:58.531697 4395 generic.go:356] "Generic (PLEG): container finished" podID="d7540c40-08d3-4c7c-a2b7-be2b32c75986" containerID="6a0eecf5b825aeca04d8d101f2efed10306d52ae715cc0ac93235785d1f28143" exitCode=0 Nov 04 04:49:58.531838 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:49:58.531792 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-546b6bbf6-ltkh7" event={"ID":"d7540c40-08d3-4c7c-a2b7-be2b32c75986","Type":"ContainerDied","Data":"6a0eecf5b825aeca04d8d101f2efed10306d52ae715cc0ac93235785d1f28143"} Nov 04 04:50:00.542861 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:00.542826 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-546b6bbf6-ltkh7" event={"ID":"d7540c40-08d3-4c7c-a2b7-be2b32c75986","Type":"ContainerStarted","Data":"89f97775b88e9c3da445c0daad799fe6e9cb3d21056b9c6b80112b4ea8b466d8"} Nov 04 04:50:00.543248 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:00.542968 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-546b6bbf6-ltkh7" Nov 04 04:50:00.812996 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:00.812947 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-546b6bbf6-ltkh7" podStartSLOduration=3.42505036 podStartE2EDuration="9.812933569s" totalImagesPullingTime="5.943661684s" totalInitContainerRuntime="444.221525ms" isStatefulPod=false podCreationTimestamp="2026-11-04 04:49:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:50:00.812063881 +0000 UTC m=+242.502146343" watchObservedRunningTime="2026-11-04 04:50:00.812933569 +0000 UTC m=+242.503016016" Nov 04 04:50:03.456248 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:50:03.456213 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 04:50:05.564101 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:05.564065 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-x4jsm" event={"ID":"43063223-b5c0-4618-8d01-6a8dde7bc38c","Type":"ContainerStarted","Data":"91052034eb9e0ee43da418bc8895ae0255ccaec6a01df0ca82c9c749863c2a65"} Nov 04 04:50:05.678094 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:05.678051 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-x4jsm" podStartSLOduration=2.538497143 podStartE2EDuration="9.678039218s" totalImagesPullingTime="7.139542075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:49:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:50:05.67770781 +0000 UTC m=+247.367790253" watchObservedRunningTime="2026-11-04 04:50:05.678039218 +0000 UTC m=+247.368121666" Nov 04 04:50:11.548890 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:11.548853 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-546b6bbf6-ltkh7" Nov 04 04:50:15.525668 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:15.525636 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-85ftp"] Nov 04 04:50:15.540923 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:15.540899 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-85ftp" Nov 04 04:50:15.595875 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:15.595846 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 04 04:50:15.638059 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:15.638015 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 04 04:50:15.638059 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:15.638030 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 04 04:50:15.638228 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:15.638172 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-pchkp\"" Nov 04 04:50:15.639553 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:15.639528 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-85ftp"] Nov 04 04:50:15.677285 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:15.677266 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhlrg\" (UniqueName: \"kubernetes.io/projected/8977878c-4c57-453c-9dab-428bd7d14140-kube-api-access-hhlrg\") pod \"dns-operator-controller-manager-8557cbdb84-85ftp\" (UID: \"8977878c-4c57-453c-9dab-428bd7d14140\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-85ftp" Nov 04 04:50:15.778355 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:15.778282 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhlrg\" (UniqueName: \"kubernetes.io/projected/8977878c-4c57-453c-9dab-428bd7d14140-kube-api-access-hhlrg\") pod \"dns-operator-controller-manager-8557cbdb84-85ftp\" (UID: \"8977878c-4c57-453c-9dab-428bd7d14140\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-85ftp" Nov 04 04:50:15.841256 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:15.841225 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhlrg\" (UniqueName: \"kubernetes.io/projected/8977878c-4c57-453c-9dab-428bd7d14140-kube-api-access-hhlrg\") pod \"dns-operator-controller-manager-8557cbdb84-85ftp\" (UID: \"8977878c-4c57-453c-9dab-428bd7d14140\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-85ftp" Nov 04 04:50:15.868947 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:15.868927 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-85ftp" Nov 04 04:50:16.263583 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:16.263562 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-85ftp"] Nov 04 04:50:16.265654 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:50:16.265621 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8977878c_4c57_453c_9dab_428bd7d14140.slice/crio-a72433ee2989b1004e3c14d28b6553371bd4039a63bd573a933553ee05414890 WatchSource:0}: Error finding container a72433ee2989b1004e3c14d28b6553371bd4039a63bd573a933553ee05414890: Status 404 returned error can't find the container with id a72433ee2989b1004e3c14d28b6553371bd4039a63bd573a933553ee05414890 Nov 04 04:50:16.610632 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:16.610507 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-85ftp" event={"ID":"8977878c-4c57-453c-9dab-428bd7d14140","Type":"ContainerStarted","Data":"a72433ee2989b1004e3c14d28b6553371bd4039a63bd573a933553ee05414890"} Nov 04 04:50:17.109568 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:17.109542 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-k4q9l"] Nov 04 04:50:17.112112 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:17.112096 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-k4q9l" Nov 04 04:50:17.162018 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:17.161979 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 04 04:50:17.162018 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:17.161984 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 04 04:50:17.188354 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:17.188330 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmj6t\" (UniqueName: \"kubernetes.io/projected/7cfc82e1-1fb4-4203-bbeb-9afddcb00041-kube-api-access-rmj6t\") pod \"cert-manager-cainjector-5c57bbd4fc-k4q9l\" (UID: \"7cfc82e1-1fb4-4203-bbeb-9afddcb00041\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-k4q9l" Nov 04 04:50:17.188475 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:17.188366 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7cfc82e1-1fb4-4203-bbeb-9afddcb00041-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-k4q9l\" (UID: \"7cfc82e1-1fb4-4203-bbeb-9afddcb00041\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-k4q9l" Nov 04 04:50:17.208995 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:17.208973 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-k4q9l"] Nov 04 04:50:17.213034 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:17.213020 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-jvk44\"" Nov 04 04:50:17.288911 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:17.288883 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmj6t\" (UniqueName: \"kubernetes.io/projected/7cfc82e1-1fb4-4203-bbeb-9afddcb00041-kube-api-access-rmj6t\") pod \"cert-manager-cainjector-5c57bbd4fc-k4q9l\" (UID: \"7cfc82e1-1fb4-4203-bbeb-9afddcb00041\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-k4q9l" Nov 04 04:50:17.289016 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:17.288918 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7cfc82e1-1fb4-4203-bbeb-9afddcb00041-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-k4q9l\" (UID: \"7cfc82e1-1fb4-4203-bbeb-9afddcb00041\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-k4q9l" Nov 04 04:50:17.346427 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:17.346401 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7cfc82e1-1fb4-4203-bbeb-9afddcb00041-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-k4q9l\" (UID: \"7cfc82e1-1fb4-4203-bbeb-9afddcb00041\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-k4q9l" Nov 04 04:50:17.346520 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:17.346463 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmj6t\" (UniqueName: \"kubernetes.io/projected/7cfc82e1-1fb4-4203-bbeb-9afddcb00041-kube-api-access-rmj6t\") pod \"cert-manager-cainjector-5c57bbd4fc-k4q9l\" (UID: \"7cfc82e1-1fb4-4203-bbeb-9afddcb00041\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-k4q9l" Nov 04 04:50:17.425973 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:17.425921 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-k4q9l" Nov 04 04:50:17.799765 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:17.799744 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-k4q9l"] Nov 04 04:50:17.801882 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:50:17.801845 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7cfc82e1_1fb4_4203_bbeb_9afddcb00041.slice/crio-21bc924fb519036d8222c5f635e0a28f1157e1d5db919a5c9f99370b4fdc0754 WatchSource:0}: Error finding container 21bc924fb519036d8222c5f635e0a28f1157e1d5db919a5c9f99370b4fdc0754: Status 404 returned error can't find the container with id 21bc924fb519036d8222c5f635e0a28f1157e1d5db919a5c9f99370b4fdc0754 Nov 04 04:50:18.621040 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:18.621000 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-k4q9l" event={"ID":"7cfc82e1-1fb4-4203-bbeb-9afddcb00041","Type":"ContainerStarted","Data":"21bc924fb519036d8222c5f635e0a28f1157e1d5db919a5c9f99370b4fdc0754"} Nov 04 04:50:20.649436 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:20.649411 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-l4g9g"] Nov 04 04:50:20.652011 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:20.651990 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-l4g9g" Nov 04 04:50:20.711437 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:20.711413 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-l4g9g"] Nov 04 04:50:20.766104 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:20.766084 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-hjlb2\"" Nov 04 04:50:20.813023 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:20.812994 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5d87\" (UniqueName: \"kubernetes.io/projected/cb487f1d-6743-4c35-ac5a-bb9ffe2f3ea3-kube-api-access-r5d87\") pod \"cert-manager-webhook-5dbbbf75cc-l4g9g\" (UID: \"cb487f1d-6743-4c35-ac5a-bb9ffe2f3ea3\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-l4g9g" Nov 04 04:50:20.813131 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:20.813029 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb487f1d-6743-4c35-ac5a-bb9ffe2f3ea3-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-l4g9g\" (UID: \"cb487f1d-6743-4c35-ac5a-bb9ffe2f3ea3\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-l4g9g" Nov 04 04:50:20.913567 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:20.913509 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb487f1d-6743-4c35-ac5a-bb9ffe2f3ea3-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-l4g9g\" (UID: \"cb487f1d-6743-4c35-ac5a-bb9ffe2f3ea3\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-l4g9g" Nov 04 04:50:20.913678 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:20.913600 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5d87\" (UniqueName: \"kubernetes.io/projected/cb487f1d-6743-4c35-ac5a-bb9ffe2f3ea3-kube-api-access-r5d87\") pod \"cert-manager-webhook-5dbbbf75cc-l4g9g\" (UID: \"cb487f1d-6743-4c35-ac5a-bb9ffe2f3ea3\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-l4g9g" Nov 04 04:50:20.970849 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:20.970824 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb487f1d-6743-4c35-ac5a-bb9ffe2f3ea3-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-l4g9g\" (UID: \"cb487f1d-6743-4c35-ac5a-bb9ffe2f3ea3\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-l4g9g" Nov 04 04:50:20.971214 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:20.971189 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5d87\" (UniqueName: \"kubernetes.io/projected/cb487f1d-6743-4c35-ac5a-bb9ffe2f3ea3-kube-api-access-r5d87\") pod \"cert-manager-webhook-5dbbbf75cc-l4g9g\" (UID: \"cb487f1d-6743-4c35-ac5a-bb9ffe2f3ea3\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-l4g9g" Nov 04 04:50:21.264620 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:21.264551 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-l4g9g" Nov 04 04:50:21.636499 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:21.636460 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-85ftp" event={"ID":"8977878c-4c57-453c-9dab-428bd7d14140","Type":"ContainerStarted","Data":"cf32f34c59f62a21fcf82dda13cbf8fedc2ccc3b2c087a8a54d8f4f110ac812b"} Nov 04 04:50:21.636697 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:21.636587 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-85ftp" Nov 04 04:50:21.644447 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:21.644362 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-l4g9g"] Nov 04 04:50:21.646863 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:50:21.646833 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb487f1d_6743_4c35_ac5a_bb9ffe2f3ea3.slice/crio-d8053d9496c848855aa14c433d9e603bdef3bb429de9f7edeb0e204f11ff6fa3 WatchSource:0}: Error finding container d8053d9496c848855aa14c433d9e603bdef3bb429de9f7edeb0e204f11ff6fa3: Status 404 returned error can't find the container with id d8053d9496c848855aa14c433d9e603bdef3bb429de9f7edeb0e204f11ff6fa3 Nov 04 04:50:21.748649 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:21.748607 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-85ftp" podStartSLOduration=2.438329995 podStartE2EDuration="6.748594177s" totalImagesPullingTime="4.310264182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:50:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:50:21.748205288 +0000 UTC m=+263.438287735" watchObservedRunningTime="2026-11-04 04:50:21.748594177 +0000 UTC m=+263.438676623" Nov 04 04:50:22.642739 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:22.642696 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-l4g9g" event={"ID":"cb487f1d-6743-4c35-ac5a-bb9ffe2f3ea3","Type":"ContainerStarted","Data":"d8053d9496c848855aa14c433d9e603bdef3bb429de9f7edeb0e204f11ff6fa3"} Nov 04 04:50:23.649365 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:23.649321 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-k4q9l" event={"ID":"7cfc82e1-1fb4-4203-bbeb-9afddcb00041","Type":"ContainerStarted","Data":"63d46b290457d238b32fdaf629336b6870fed58af43f0490bff011c46fdd73bb"} Nov 04 04:50:23.651310 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:23.651268 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-l4g9g" event={"ID":"cb487f1d-6743-4c35-ac5a-bb9ffe2f3ea3","Type":"ContainerStarted","Data":"4d4ebd574af5f9e57d0ea5680adb698805c41250b765ba63523740eb53329951"} Nov 04 04:50:23.651435 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:23.651377 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-l4g9g" Nov 04 04:50:23.786596 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:23.786503 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-k4q9l" podStartSLOduration=1.703542363 podStartE2EDuration="6.786489311s" totalImagesPullingTime="5.082946948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:50:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:50:23.784129462 +0000 UTC m=+265.474211899" watchObservedRunningTime="2026-11-04 04:50:23.786489311 +0000 UTC m=+265.476571749" Nov 04 04:50:23.899588 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:23.899541 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-l4g9g" podStartSLOduration=2.136243173 podStartE2EDuration="3.899525362s" totalImagesPullingTime="1.763282189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:50:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:50:23.898537595 +0000 UTC m=+265.588620042" watchObservedRunningTime="2026-11-04 04:50:23.899525362 +0000 UTC m=+265.589607808" Nov 04 04:50:25.681891 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:25.681857 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-655d896fb8-qsdbq"] Nov 04 04:50:25.684483 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:25.684456 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-655d896fb8-qsdbq" Nov 04 04:50:25.733747 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:25.733723 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 04 04:50:25.741567 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:25.741547 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-txlxj\"" Nov 04 04:50:25.743762 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:25.743744 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 04 04:50:25.781464 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:25.781442 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-655d896fb8-qsdbq"] Nov 04 04:50:25.782777 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:25.782761 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-z7rgw"] Nov 04 04:50:25.784924 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:25.784909 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-z7rgw" Nov 04 04:50:25.833684 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:25.833658 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-z7rgw"] Nov 04 04:50:25.837871 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:25.837854 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-8t897\"" Nov 04 04:50:25.855361 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:25.855343 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcc5b454-668a-4330-9f05-3e1c256bf81e-tls-certs\") pod \"kube-auth-proxy-655d896fb8-qsdbq\" (UID: \"dcc5b454-668a-4330-9f05-3e1c256bf81e\") " pod="openshift-ingress/kube-auth-proxy-655d896fb8-qsdbq" Nov 04 04:50:25.855442 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:25.855385 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkpwk\" (UniqueName: \"kubernetes.io/projected/dcc5b454-668a-4330-9f05-3e1c256bf81e-kube-api-access-dkpwk\") pod \"kube-auth-proxy-655d896fb8-qsdbq\" (UID: \"dcc5b454-668a-4330-9f05-3e1c256bf81e\") " pod="openshift-ingress/kube-auth-proxy-655d896fb8-qsdbq" Nov 04 04:50:25.855481 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:25.855459 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dcc5b454-668a-4330-9f05-3e1c256bf81e-tmp\") pod \"kube-auth-proxy-655d896fb8-qsdbq\" (UID: \"dcc5b454-668a-4330-9f05-3e1c256bf81e\") " pod="openshift-ingress/kube-auth-proxy-655d896fb8-qsdbq" Nov 04 04:50:25.956717 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:25.956665 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c29a1e61-1593-4a4a-a7b2-80e58875a44f-bound-sa-token\") pod \"cert-manager-597b95cb84-z7rgw\" (UID: \"c29a1e61-1593-4a4a-a7b2-80e58875a44f\") " pod="cert-manager/cert-manager-597b95cb84-z7rgw" Nov 04 04:50:25.956717 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:25.956702 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcc5b454-668a-4330-9f05-3e1c256bf81e-tls-certs\") pod \"kube-auth-proxy-655d896fb8-qsdbq\" (UID: \"dcc5b454-668a-4330-9f05-3e1c256bf81e\") " pod="openshift-ingress/kube-auth-proxy-655d896fb8-qsdbq" Nov 04 04:50:25.956867 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:25.956719 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkpwk\" (UniqueName: \"kubernetes.io/projected/dcc5b454-668a-4330-9f05-3e1c256bf81e-kube-api-access-dkpwk\") pod \"kube-auth-proxy-655d896fb8-qsdbq\" (UID: \"dcc5b454-668a-4330-9f05-3e1c256bf81e\") " pod="openshift-ingress/kube-auth-proxy-655d896fb8-qsdbq" Nov 04 04:50:25.956867 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:25.956779 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dcc5b454-668a-4330-9f05-3e1c256bf81e-tmp\") pod \"kube-auth-proxy-655d896fb8-qsdbq\" (UID: \"dcc5b454-668a-4330-9f05-3e1c256bf81e\") " pod="openshift-ingress/kube-auth-proxy-655d896fb8-qsdbq" Nov 04 04:50:25.956867 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:25.956816 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lptgs\" (UniqueName: \"kubernetes.io/projected/c29a1e61-1593-4a4a-a7b2-80e58875a44f-kube-api-access-lptgs\") pod \"cert-manager-597b95cb84-z7rgw\" (UID: \"c29a1e61-1593-4a4a-a7b2-80e58875a44f\") " pod="cert-manager/cert-manager-597b95cb84-z7rgw" Nov 04 04:50:25.958931 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:25.958907 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dcc5b454-668a-4330-9f05-3e1c256bf81e-tmp\") pod \"kube-auth-proxy-655d896fb8-qsdbq\" (UID: \"dcc5b454-668a-4330-9f05-3e1c256bf81e\") " pod="openshift-ingress/kube-auth-proxy-655d896fb8-qsdbq" Nov 04 04:50:25.959035 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:25.959016 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcc5b454-668a-4330-9f05-3e1c256bf81e-tls-certs\") pod \"kube-auth-proxy-655d896fb8-qsdbq\" (UID: \"dcc5b454-668a-4330-9f05-3e1c256bf81e\") " pod="openshift-ingress/kube-auth-proxy-655d896fb8-qsdbq" Nov 04 04:50:26.014845 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:26.014822 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkpwk\" (UniqueName: \"kubernetes.io/projected/dcc5b454-668a-4330-9f05-3e1c256bf81e-kube-api-access-dkpwk\") pod \"kube-auth-proxy-655d896fb8-qsdbq\" (UID: \"dcc5b454-668a-4330-9f05-3e1c256bf81e\") " pod="openshift-ingress/kube-auth-proxy-655d896fb8-qsdbq" Nov 04 04:50:26.057477 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:26.057458 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c29a1e61-1593-4a4a-a7b2-80e58875a44f-bound-sa-token\") pod \"cert-manager-597b95cb84-z7rgw\" (UID: \"c29a1e61-1593-4a4a-a7b2-80e58875a44f\") " pod="cert-manager/cert-manager-597b95cb84-z7rgw" Nov 04 04:50:26.057565 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:26.057499 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lptgs\" (UniqueName: \"kubernetes.io/projected/c29a1e61-1593-4a4a-a7b2-80e58875a44f-kube-api-access-lptgs\") pod \"cert-manager-597b95cb84-z7rgw\" (UID: \"c29a1e61-1593-4a4a-a7b2-80e58875a44f\") " pod="cert-manager/cert-manager-597b95cb84-z7rgw" Nov 04 04:50:26.123512 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:26.123486 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lptgs\" (UniqueName: \"kubernetes.io/projected/c29a1e61-1593-4a4a-a7b2-80e58875a44f-kube-api-access-lptgs\") pod \"cert-manager-597b95cb84-z7rgw\" (UID: \"c29a1e61-1593-4a4a-a7b2-80e58875a44f\") " pod="cert-manager/cert-manager-597b95cb84-z7rgw" Nov 04 04:50:26.124416 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:26.124398 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c29a1e61-1593-4a4a-a7b2-80e58875a44f-bound-sa-token\") pod \"cert-manager-597b95cb84-z7rgw\" (UID: \"c29a1e61-1593-4a4a-a7b2-80e58875a44f\") " pod="cert-manager/cert-manager-597b95cb84-z7rgw" Nov 04 04:50:26.294550 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:26.294530 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-655d896fb8-qsdbq" Nov 04 04:50:26.397215 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:26.397185 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-z7rgw" Nov 04 04:50:26.663741 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:26.663721 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-655d896fb8-qsdbq"] Nov 04 04:50:26.666139 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:50:26.666102 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcc5b454_668a_4330_9f05_3e1c256bf81e.slice/crio-ee661db26cf500181796b2c59b3dbed1e29b26d4d7917f9e2ef8c7a34a170c64 WatchSource:0}: Error finding container ee661db26cf500181796b2c59b3dbed1e29b26d4d7917f9e2ef8c7a34a170c64: Status 404 returned error can't find the container with id ee661db26cf500181796b2c59b3dbed1e29b26d4d7917f9e2ef8c7a34a170c64 Nov 04 04:50:26.769148 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:26.769126 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-z7rgw"] Nov 04 04:50:26.770984 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:50:26.770955 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc29a1e61_1593_4a4a_a7b2_80e58875a44f.slice/crio-5b4e07850a618a013e3fd08cad0c79f75fa4e1e4763ec1a3b9fbffaafbda5c2d WatchSource:0}: Error finding container 5b4e07850a618a013e3fd08cad0c79f75fa4e1e4763ec1a3b9fbffaafbda5c2d: Status 404 returned error can't find the container with id 5b4e07850a618a013e3fd08cad0c79f75fa4e1e4763ec1a3b9fbffaafbda5c2d Nov 04 04:50:27.671166 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:27.671130 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-z7rgw" event={"ID":"c29a1e61-1593-4a4a-a7b2-80e58875a44f","Type":"ContainerStarted","Data":"772fc44984623a1edf7fb9034916536f48d7ed597e6c932d9daf9cc7209b2d3e"} Nov 04 04:50:27.671356 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:27.671178 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-z7rgw" event={"ID":"c29a1e61-1593-4a4a-a7b2-80e58875a44f","Type":"ContainerStarted","Data":"5b4e07850a618a013e3fd08cad0c79f75fa4e1e4763ec1a3b9fbffaafbda5c2d"} Nov 04 04:50:27.672343 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:27.672322 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-655d896fb8-qsdbq" event={"ID":"dcc5b454-668a-4330-9f05-3e1c256bf81e","Type":"ContainerStarted","Data":"ee661db26cf500181796b2c59b3dbed1e29b26d4d7917f9e2ef8c7a34a170c64"} Nov 04 04:50:27.780681 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:27.780638 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-z7rgw" podStartSLOduration=2.780625542 podStartE2EDuration="2.780625542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:50:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:50:27.779475246 +0000 UTC m=+269.469557694" watchObservedRunningTime="2026-11-04 04:50:27.780625542 +0000 UTC m=+269.470707988" Nov 04 04:50:29.657513 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:29.657483 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-l4g9g" Nov 04 04:50:30.687462 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:30.687437 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-655d896fb8-qsdbq" event={"ID":"dcc5b454-668a-4330-9f05-3e1c256bf81e","Type":"ContainerStarted","Data":"16e075465cbccea80dfa12e832301829ee2a251be19e88b3d4a14155eb4ed4b2"} Nov 04 04:50:30.796790 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:30.796745 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-655d896fb8-qsdbq" podStartSLOduration=1.8388110050000002 podStartE2EDuration="5.796731443s" totalImagesPullingTime="3.957920438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:50:30.795347736 +0000 UTC m=+272.485430182" watchObservedRunningTime="2026-11-04 04:50:30.796731443 +0000 UTC m=+272.486813890" Nov 04 04:50:32.600404 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:32.600360 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5g2bm"] Nov 04 04:50:32.604198 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:32.604173 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5g2bm" Nov 04 04:50:32.645389 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:32.645362 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-85ftp" Nov 04 04:50:32.653194 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:32.653171 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-4gq99\"" Nov 04 04:50:32.700857 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:32.700832 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5g2bm"] Nov 04 04:50:32.707876 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:32.707858 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m85mq\" (UniqueName: \"kubernetes.io/projected/846fda69-90d5-463f-b306-40b443e8fbb7-kube-api-access-m85mq\") pod \"kuadrant-operator-controller-manager-678d87766f-5g2bm\" (UID: \"846fda69-90d5-463f-b306-40b443e8fbb7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5g2bm" Nov 04 04:50:32.707951 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:32.707890 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/846fda69-90d5-463f-b306-40b443e8fbb7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-5g2bm\" (UID: \"846fda69-90d5-463f-b306-40b443e8fbb7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5g2bm" Nov 04 04:50:32.808406 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:32.808383 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m85mq\" (UniqueName: \"kubernetes.io/projected/846fda69-90d5-463f-b306-40b443e8fbb7-kube-api-access-m85mq\") pod \"kuadrant-operator-controller-manager-678d87766f-5g2bm\" (UID: \"846fda69-90d5-463f-b306-40b443e8fbb7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5g2bm" Nov 04 04:50:32.808518 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:32.808419 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/846fda69-90d5-463f-b306-40b443e8fbb7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-5g2bm\" (UID: \"846fda69-90d5-463f-b306-40b443e8fbb7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5g2bm" Nov 04 04:50:32.808730 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:32.808713 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/846fda69-90d5-463f-b306-40b443e8fbb7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-5g2bm\" (UID: \"846fda69-90d5-463f-b306-40b443e8fbb7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5g2bm" Nov 04 04:50:32.872266 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:32.872203 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m85mq\" (UniqueName: \"kubernetes.io/projected/846fda69-90d5-463f-b306-40b443e8fbb7-kube-api-access-m85mq\") pod \"kuadrant-operator-controller-manager-678d87766f-5g2bm\" (UID: \"846fda69-90d5-463f-b306-40b443e8fbb7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5g2bm" Nov 04 04:50:32.917695 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:32.917661 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5g2bm" Nov 04 04:50:33.346516 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:33.346491 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5g2bm"] Nov 04 04:50:33.349747 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:50:33.349719 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod846fda69_90d5_463f_b306_40b443e8fbb7.slice/crio-b817ce2eb54bd9664b9f314d6461a3cb432dff3dae34ae422e9c30735fe530a0 WatchSource:0}: Error finding container b817ce2eb54bd9664b9f314d6461a3cb432dff3dae34ae422e9c30735fe530a0: Status 404 returned error can't find the container with id b817ce2eb54bd9664b9f314d6461a3cb432dff3dae34ae422e9c30735fe530a0 Nov 04 04:50:33.456085 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:50:33.456052 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 04:50:33.702674 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:33.702603 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5g2bm" event={"ID":"846fda69-90d5-463f-b306-40b443e8fbb7","Type":"ContainerStarted","Data":"b817ce2eb54bd9664b9f314d6461a3cb432dff3dae34ae422e9c30735fe530a0"} Nov 04 04:50:43.747539 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:43.747499 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5g2bm" event={"ID":"846fda69-90d5-463f-b306-40b443e8fbb7","Type":"ContainerStarted","Data":"3fb6cc3bf70f046dde1ec95797e973706e0878bece49575d3745ff46cd355a9c"} Nov 04 04:50:43.747957 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:43.747563 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5g2bm" Nov 04 04:50:43.951964 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:43.951916 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5g2bm" podStartSLOduration=2.59788882 podStartE2EDuration="11.95190373s" totalImagesPullingTime="9.35401491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:50:43.905177809 +0000 UTC m=+285.595260257" watchObservedRunningTime="2026-11-04 04:50:43.95190373 +0000 UTC m=+285.641986176" Nov 04 04:50:50.645312 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:50.645264 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-gtgkr"] Nov 04 04:50:50.648954 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:50.648931 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-gtgkr" Nov 04 04:50:50.697597 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:50.697573 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6gwpp\"" Nov 04 04:50:50.743529 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:50.743499 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtrr4\" (UniqueName: \"kubernetes.io/projected/1fdc34d3-b6a1-479e-9c7c-f416774a2f8b-kube-api-access-gtrr4\") pod \"authorino-57f7f8659f-gtgkr\" (UID: \"1fdc34d3-b6a1-479e-9c7c-f416774a2f8b\") " pod="kuadrant-system/authorino-57f7f8659f-gtgkr" Nov 04 04:50:50.745445 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:50.745426 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-gtgkr"] Nov 04 04:50:50.844591 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:50.844570 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtrr4\" (UniqueName: \"kubernetes.io/projected/1fdc34d3-b6a1-479e-9c7c-f416774a2f8b-kube-api-access-gtrr4\") pod \"authorino-57f7f8659f-gtgkr\" (UID: \"1fdc34d3-b6a1-479e-9c7c-f416774a2f8b\") " pod="kuadrant-system/authorino-57f7f8659f-gtgkr" Nov 04 04:50:50.902184 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:50.902135 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtrr4\" (UniqueName: \"kubernetes.io/projected/1fdc34d3-b6a1-479e-9c7c-f416774a2f8b-kube-api-access-gtrr4\") pod \"authorino-57f7f8659f-gtgkr\" (UID: \"1fdc34d3-b6a1-479e-9c7c-f416774a2f8b\") " pod="kuadrant-system/authorino-57f7f8659f-gtgkr" Nov 04 04:50:50.961052 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:50.961030 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-gtgkr" Nov 04 04:50:51.332139 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:51.332116 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-gtgkr"] Nov 04 04:50:51.334474 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:50:51.334452 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fdc34d3_b6a1_479e_9c7c_f416774a2f8b.slice/crio-961e325a0594eddd52cdb2307d111b5aa727b49412e63f24a46d93eb9143fcd3 WatchSource:0}: Error finding container 961e325a0594eddd52cdb2307d111b5aa727b49412e63f24a46d93eb9143fcd3: Status 404 returned error can't find the container with id 961e325a0594eddd52cdb2307d111b5aa727b49412e63f24a46d93eb9143fcd3 Nov 04 04:50:51.783652 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:51.783585 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-gtgkr" event={"ID":"1fdc34d3-b6a1-479e-9c7c-f416774a2f8b","Type":"ContainerStarted","Data":"961e325a0594eddd52cdb2307d111b5aa727b49412e63f24a46d93eb9143fcd3"} Nov 04 04:50:54.754157 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:54.754125 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-5g2bm" Nov 04 04:50:56.363375 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:56.363346 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-gtgkr"] Nov 04 04:50:56.809435 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:56.809404 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-gtgkr" event={"ID":"1fdc34d3-b6a1-479e-9c7c-f416774a2f8b","Type":"ContainerStarted","Data":"adc64541bd9dc7b726e8701cb93bfc0ff989aa4c41ebf2dcdcee354858595b76"} Nov 04 04:50:56.916407 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:56.916362 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-gtgkr" podStartSLOduration=2.077532666 podStartE2EDuration="6.916350357s" totalImagesPullingTime="4.838817691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:50:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:50:56.915969885 +0000 UTC m=+298.606052332" watchObservedRunningTime="2026-11-04 04:50:56.916350357 +0000 UTC m=+298.606432803" Nov 04 04:50:57.814290 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:57.814249 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-gtgkr" podUID="1fdc34d3-b6a1-479e-9c7c-f416774a2f8b" containerName="authorino" containerID="cri-o://adc64541bd9dc7b726e8701cb93bfc0ff989aa4c41ebf2dcdcee354858595b76" gracePeriod=30 Nov 04 04:50:58.048545 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:58.048515 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-gtgkr" Nov 04 04:50:58.092559 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:58.092502 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fdc34d3-b6a1-479e-9c7c-f416774a2f8b-kube-api-access-gtrr4\" (UniqueName: \"kubernetes.io/projected/1fdc34d3-b6a1-479e-9c7c-f416774a2f8b-kube-api-access-gtrr4\") pod \"1fdc34d3-b6a1-479e-9c7c-f416774a2f8b\" (UID: \"1fdc34d3-b6a1-479e-9c7c-f416774a2f8b\") " Nov 04 04:50:58.094407 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:58.094381 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fdc34d3-b6a1-479e-9c7c-f416774a2f8b-kube-api-access-gtrr4" pod "1fdc34d3-b6a1-479e-9c7c-f416774a2f8b" (UID: "1fdc34d3-b6a1-479e-9c7c-f416774a2f8b"). InnerVolumeSpecName "kube-api-access-gtrr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:50:58.193413 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:58.193387 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtrr4\" (UniqueName: \"kubernetes.io/projected/1fdc34d3-b6a1-479e-9c7c-f416774a2f8b-kube-api-access-gtrr4\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:50:58.820497 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:58.820456 4395 generic.go:356] "Generic (PLEG): container finished" podID="1fdc34d3-b6a1-479e-9c7c-f416774a2f8b" containerID="adc64541bd9dc7b726e8701cb93bfc0ff989aa4c41ebf2dcdcee354858595b76" exitCode=0 Nov 04 04:50:58.820923 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:58.820501 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-gtgkr" Nov 04 04:50:58.820923 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:58.820547 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-gtgkr" event={"ID":"1fdc34d3-b6a1-479e-9c7c-f416774a2f8b","Type":"ContainerDied","Data":"adc64541bd9dc7b726e8701cb93bfc0ff989aa4c41ebf2dcdcee354858595b76"} Nov 04 04:50:58.820923 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:58.820602 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-gtgkr" event={"ID":"1fdc34d3-b6a1-479e-9c7c-f416774a2f8b","Type":"ContainerDied","Data":"961e325a0594eddd52cdb2307d111b5aa727b49412e63f24a46d93eb9143fcd3"} Nov 04 04:50:58.820923 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:58.820621 4395 scope.go:122] "RemoveContainer" containerID="adc64541bd9dc7b726e8701cb93bfc0ff989aa4c41ebf2dcdcee354858595b76" Nov 04 04:50:58.829228 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:58.829209 4395 scope.go:122] "RemoveContainer" containerID="adc64541bd9dc7b726e8701cb93bfc0ff989aa4c41ebf2dcdcee354858595b76" Nov 04 04:50:58.829573 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:50:58.829552 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adc64541bd9dc7b726e8701cb93bfc0ff989aa4c41ebf2dcdcee354858595b76\": container with ID starting with adc64541bd9dc7b726e8701cb93bfc0ff989aa4c41ebf2dcdcee354858595b76 not found: ID does not exist" containerID="adc64541bd9dc7b726e8701cb93bfc0ff989aa4c41ebf2dcdcee354858595b76" Nov 04 04:50:58.829624 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:58.829581 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adc64541bd9dc7b726e8701cb93bfc0ff989aa4c41ebf2dcdcee354858595b76"} err="failed to get container status \"adc64541bd9dc7b726e8701cb93bfc0ff989aa4c41ebf2dcdcee354858595b76\": rpc error: code = NotFound desc = could not find container \"adc64541bd9dc7b726e8701cb93bfc0ff989aa4c41ebf2dcdcee354858595b76\": container with ID starting with adc64541bd9dc7b726e8701cb93bfc0ff989aa4c41ebf2dcdcee354858595b76 not found: ID does not exist" Nov 04 04:50:58.973793 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:58.973767 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-gtgkr"] Nov 04 04:50:59.019820 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:50:59.019794 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-gtgkr"] Nov 04 04:51:00.414614 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:00.414569 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fdc34d3-b6a1-479e-9c7c-f416774a2f8b" path="/var/lib/kubelet/pods/1fdc34d3-b6a1-479e-9c7c-f416774a2f8b/volumes" Nov 04 04:51:03.457971 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:51:03.457933 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 04:51:33.456216 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:51:33.456137 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 04:51:38.881812 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:38.881780 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6c844db6b-c5mnw"] Nov 04 04:51:38.882280 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:38.882125 4395 cpu_manager.go:414] "containerMap: removing container" podUID="1fdc34d3-b6a1-479e-9c7c-f416774a2f8b" containerName="authorino" Nov 04 04:51:38.882280 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:38.882139 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fdc34d3-b6a1-479e-9c7c-f416774a2f8b" containerName="authorino" Nov 04 04:51:38.882280 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:38.882212 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fdc34d3-b6a1-479e-9c7c-f416774a2f8b" containerName="authorino" Nov 04 04:51:38.886637 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:38.886611 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c844db6b-c5mnw" Nov 04 04:51:38.935624 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:38.935599 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 04 04:51:38.981700 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:38.981675 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 04 04:51:38.981844 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:38.981708 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 04 04:51:38.981844 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:38.981708 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 04 04:51:38.981844 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:38.981782 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8lbvc\"" Nov 04 04:51:38.981844 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:38.981800 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 04 04:51:38.983379 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:38.983361 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c844db6b-c5mnw"] Nov 04 04:51:38.998046 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:38.998026 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/126bfc55-f2af-4189-8ee0-3b383e4789e8-metrics-certs\") pod \"maas-controller-6c844db6b-c5mnw\" (UID: \"126bfc55-f2af-4189-8ee0-3b383e4789e8\") " pod="opendatahub/maas-controller-6c844db6b-c5mnw" Nov 04 04:51:38.998114 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:38.998067 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/126bfc55-f2af-4189-8ee0-3b383e4789e8-webhook-cert\") pod \"maas-controller-6c844db6b-c5mnw\" (UID: \"126bfc55-f2af-4189-8ee0-3b383e4789e8\") " pod="opendatahub/maas-controller-6c844db6b-c5mnw" Nov 04 04:51:38.998154 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:38.998141 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bpx9\" (UniqueName: \"kubernetes.io/projected/126bfc55-f2af-4189-8ee0-3b383e4789e8-kube-api-access-9bpx9\") pod \"maas-controller-6c844db6b-c5mnw\" (UID: \"126bfc55-f2af-4189-8ee0-3b383e4789e8\") " pod="opendatahub/maas-controller-6c844db6b-c5mnw" Nov 04 04:51:39.099241 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:39.099211 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/126bfc55-f2af-4189-8ee0-3b383e4789e8-webhook-cert\") pod \"maas-controller-6c844db6b-c5mnw\" (UID: \"126bfc55-f2af-4189-8ee0-3b383e4789e8\") " pod="opendatahub/maas-controller-6c844db6b-c5mnw" Nov 04 04:51:39.099405 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:39.099256 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bpx9\" (UniqueName: \"kubernetes.io/projected/126bfc55-f2af-4189-8ee0-3b383e4789e8-kube-api-access-9bpx9\") pod \"maas-controller-6c844db6b-c5mnw\" (UID: \"126bfc55-f2af-4189-8ee0-3b383e4789e8\") " pod="opendatahub/maas-controller-6c844db6b-c5mnw" Nov 04 04:51:39.099405 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:39.099289 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/126bfc55-f2af-4189-8ee0-3b383e4789e8-metrics-certs\") pod \"maas-controller-6c844db6b-c5mnw\" (UID: \"126bfc55-f2af-4189-8ee0-3b383e4789e8\") " pod="opendatahub/maas-controller-6c844db6b-c5mnw" Nov 04 04:51:39.101605 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:39.101580 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/126bfc55-f2af-4189-8ee0-3b383e4789e8-metrics-certs\") pod \"maas-controller-6c844db6b-c5mnw\" (UID: \"126bfc55-f2af-4189-8ee0-3b383e4789e8\") " pod="opendatahub/maas-controller-6c844db6b-c5mnw" Nov 04 04:51:39.101708 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:39.101615 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/126bfc55-f2af-4189-8ee0-3b383e4789e8-webhook-cert\") pod \"maas-controller-6c844db6b-c5mnw\" (UID: \"126bfc55-f2af-4189-8ee0-3b383e4789e8\") " pod="opendatahub/maas-controller-6c844db6b-c5mnw" Nov 04 04:51:39.158652 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:39.158599 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bpx9\" (UniqueName: \"kubernetes.io/projected/126bfc55-f2af-4189-8ee0-3b383e4789e8-kube-api-access-9bpx9\") pod \"maas-controller-6c844db6b-c5mnw\" (UID: \"126bfc55-f2af-4189-8ee0-3b383e4789e8\") " pod="opendatahub/maas-controller-6c844db6b-c5mnw" Nov 04 04:51:39.199052 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:39.199029 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c844db6b-c5mnw" Nov 04 04:51:39.571034 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:39.571014 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c844db6b-c5mnw"] Nov 04 04:51:39.573010 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:51:39.572979 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod126bfc55_f2af_4189_8ee0_3b383e4789e8.slice/crio-9eb46d144e44299a11c9ff9239b733fde8ceb774dac25480fea5b867ee5fa0b8 WatchSource:0}: Error finding container 9eb46d144e44299a11c9ff9239b733fde8ceb774dac25480fea5b867ee5fa0b8: Status 404 returned error can't find the container with id 9eb46d144e44299a11c9ff9239b733fde8ceb774dac25480fea5b867ee5fa0b8 Nov 04 04:51:39.574447 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:39.574427 4395 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 04:51:40.031261 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:40.031181 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c844db6b-c5mnw" event={"ID":"126bfc55-f2af-4189-8ee0-3b383e4789e8","Type":"ContainerStarted","Data":"9eb46d144e44299a11c9ff9239b733fde8ceb774dac25480fea5b867ee5fa0b8"} Nov 04 04:51:44.052794 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:44.052758 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c844db6b-c5mnw" event={"ID":"126bfc55-f2af-4189-8ee0-3b383e4789e8","Type":"ContainerStarted","Data":"31476d00c46d6846ba6c8d408a5873073735093a2c8ba4b1fa419f249a437087"} Nov 04 04:51:44.053164 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:44.052865 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6c844db6b-c5mnw" Nov 04 04:51:44.163333 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:44.163271 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6c844db6b-c5mnw" podStartSLOduration=2.481407268 podStartE2EDuration="6.163257091s" totalImagesPullingTime="3.681849823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:51:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:51:44.163211984 +0000 UTC m=+345.853294430" watchObservedRunningTime="2026-11-04 04:51:44.163257091 +0000 UTC m=+345.853339517" Nov 04 04:51:45.511362 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:45.511330 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-9cxmr"] Nov 04 04:51:45.514991 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:45.514968 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9cxmr" Nov 04 04:51:45.563517 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:45.563495 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-w57kc\"" Nov 04 04:51:45.563517 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:45.563496 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 04 04:51:45.611285 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:45.611261 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-9cxmr"] Nov 04 04:51:45.654777 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:45.654747 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3759482-cd05-4daf-ad79-93a2e9e1e35b-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-9cxmr\" (UID: \"a3759482-cd05-4daf-ad79-93a2e9e1e35b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9cxmr" Nov 04 04:51:45.654887 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:45.654809 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq74k\" (UniqueName: \"kubernetes.io/projected/a3759482-cd05-4daf-ad79-93a2e9e1e35b-kube-api-access-gq74k\") pod \"llmisvc-controller-manager-869d5dcc5d-9cxmr\" (UID: \"a3759482-cd05-4daf-ad79-93a2e9e1e35b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9cxmr" Nov 04 04:51:45.726375 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:45.726348 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-5gpsn"] Nov 04 04:51:45.729687 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:45.729669 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-5gpsn" Nov 04 04:51:45.756157 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:45.756137 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3759482-cd05-4daf-ad79-93a2e9e1e35b-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-9cxmr\" (UID: \"a3759482-cd05-4daf-ad79-93a2e9e1e35b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9cxmr" Nov 04 04:51:45.756232 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:45.756177 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq74k\" (UniqueName: \"kubernetes.io/projected/a3759482-cd05-4daf-ad79-93a2e9e1e35b-kube-api-access-gq74k\") pod \"llmisvc-controller-manager-869d5dcc5d-9cxmr\" (UID: \"a3759482-cd05-4daf-ad79-93a2e9e1e35b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9cxmr" Nov 04 04:51:45.758426 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:45.758404 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3759482-cd05-4daf-ad79-93a2e9e1e35b-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-9cxmr\" (UID: \"a3759482-cd05-4daf-ad79-93a2e9e1e35b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9cxmr" Nov 04 04:51:45.778502 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:45.778448 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 04 04:51:45.778502 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:45.778448 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-c8jx9\"" Nov 04 04:51:45.778629 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:45.778508 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 04 04:51:45.786555 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:45.786532 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-5gpsn"] Nov 04 04:51:45.813125 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:45.813099 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq74k\" (UniqueName: \"kubernetes.io/projected/a3759482-cd05-4daf-ad79-93a2e9e1e35b-kube-api-access-gq74k\") pod \"llmisvc-controller-manager-869d5dcc5d-9cxmr\" (UID: \"a3759482-cd05-4daf-ad79-93a2e9e1e35b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9cxmr" Nov 04 04:51:45.826659 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:45.826626 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9cxmr" Nov 04 04:51:45.857404 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:45.857383 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6efbb1d-ceb4-4612-9ed2-266f9c8b43da-metrics-certs\") pod \"odh-model-controller-769875db56-5gpsn\" (UID: \"b6efbb1d-ceb4-4612-9ed2-266f9c8b43da\") " pod="opendatahub/odh-model-controller-769875db56-5gpsn" Nov 04 04:51:45.857503 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:45.857410 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6566\" (UniqueName: \"kubernetes.io/projected/b6efbb1d-ceb4-4612-9ed2-266f9c8b43da-kube-api-access-r6566\") pod \"odh-model-controller-769875db56-5gpsn\" (UID: \"b6efbb1d-ceb4-4612-9ed2-266f9c8b43da\") " pod="opendatahub/odh-model-controller-769875db56-5gpsn" Nov 04 04:51:45.857503 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:45.857455 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b6efbb1d-ceb4-4612-9ed2-266f9c8b43da-cert\") pod \"odh-model-controller-769875db56-5gpsn\" (UID: \"b6efbb1d-ceb4-4612-9ed2-266f9c8b43da\") " pod="opendatahub/odh-model-controller-769875db56-5gpsn" Nov 04 04:51:45.958454 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:45.958420 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6efbb1d-ceb4-4612-9ed2-266f9c8b43da-metrics-certs\") pod \"odh-model-controller-769875db56-5gpsn\" (UID: \"b6efbb1d-ceb4-4612-9ed2-266f9c8b43da\") " pod="opendatahub/odh-model-controller-769875db56-5gpsn" Nov 04 04:51:45.958591 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:45.958461 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6566\" (UniqueName: \"kubernetes.io/projected/b6efbb1d-ceb4-4612-9ed2-266f9c8b43da-kube-api-access-r6566\") pod \"odh-model-controller-769875db56-5gpsn\" (UID: \"b6efbb1d-ceb4-4612-9ed2-266f9c8b43da\") " pod="opendatahub/odh-model-controller-769875db56-5gpsn" Nov 04 04:51:45.958591 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:45.958515 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b6efbb1d-ceb4-4612-9ed2-266f9c8b43da-cert\") pod \"odh-model-controller-769875db56-5gpsn\" (UID: \"b6efbb1d-ceb4-4612-9ed2-266f9c8b43da\") " pod="opendatahub/odh-model-controller-769875db56-5gpsn" Nov 04 04:51:45.960700 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:45.960673 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6efbb1d-ceb4-4612-9ed2-266f9c8b43da-metrics-certs\") pod \"odh-model-controller-769875db56-5gpsn\" (UID: \"b6efbb1d-ceb4-4612-9ed2-266f9c8b43da\") " pod="opendatahub/odh-model-controller-769875db56-5gpsn" Nov 04 04:51:45.960817 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:45.960740 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b6efbb1d-ceb4-4612-9ed2-266f9c8b43da-cert\") pod \"odh-model-controller-769875db56-5gpsn\" (UID: \"b6efbb1d-ceb4-4612-9ed2-266f9c8b43da\") " pod="opendatahub/odh-model-controller-769875db56-5gpsn" Nov 04 04:51:46.015560 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:46.015537 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6566\" (UniqueName: \"kubernetes.io/projected/b6efbb1d-ceb4-4612-9ed2-266f9c8b43da-kube-api-access-r6566\") pod \"odh-model-controller-769875db56-5gpsn\" (UID: \"b6efbb1d-ceb4-4612-9ed2-266f9c8b43da\") " pod="opendatahub/odh-model-controller-769875db56-5gpsn" Nov 04 04:51:46.042436 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:46.042383 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-5gpsn" Nov 04 04:51:46.199393 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:46.199368 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-9cxmr"] Nov 04 04:51:46.201145 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:51:46.201125 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda3759482_cd05_4daf_ad79_93a2e9e1e35b.slice/crio-866983afa324bf680c558dbec37ba56d51dbe89d72d0a8a900215bdfbd3790e5 WatchSource:0}: Error finding container 866983afa324bf680c558dbec37ba56d51dbe89d72d0a8a900215bdfbd3790e5: Status 404 returned error can't find the container with id 866983afa324bf680c558dbec37ba56d51dbe89d72d0a8a900215bdfbd3790e5 Nov 04 04:51:46.415206 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:46.415182 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-5gpsn"] Nov 04 04:51:46.416167 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:51:46.416138 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6efbb1d_ceb4_4612_9ed2_266f9c8b43da.slice/crio-ecdead84168eea048ee3faf4315fbb2250c1e39782ea78f36f0979f1c1606bee WatchSource:0}: Error finding container ecdead84168eea048ee3faf4315fbb2250c1e39782ea78f36f0979f1c1606bee: Status 404 returned error can't find the container with id ecdead84168eea048ee3faf4315fbb2250c1e39782ea78f36f0979f1c1606bee Nov 04 04:51:47.069052 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:47.069012 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-5gpsn" event={"ID":"b6efbb1d-ceb4-4612-9ed2-266f9c8b43da","Type":"ContainerStarted","Data":"ecdead84168eea048ee3faf4315fbb2250c1e39782ea78f36f0979f1c1606bee"} Nov 04 04:51:47.070219 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:47.070197 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9cxmr" event={"ID":"a3759482-cd05-4daf-ad79-93a2e9e1e35b","Type":"ContainerStarted","Data":"866983afa324bf680c558dbec37ba56d51dbe89d72d0a8a900215bdfbd3790e5"} Nov 04 04:51:51.094028 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:51.093986 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9cxmr" event={"ID":"a3759482-cd05-4daf-ad79-93a2e9e1e35b","Type":"ContainerStarted","Data":"2ec2ec37f907212571dec987531f3fd0100828cc81cd5c18b70e7d5c889cce0b"} Nov 04 04:51:51.094355 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:51.094058 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9cxmr" Nov 04 04:51:51.095567 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:51.095541 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-5gpsn" event={"ID":"b6efbb1d-ceb4-4612-9ed2-266f9c8b43da","Type":"ContainerStarted","Data":"f4709b716799c17dc2e68f5b834e01ea179794828284f00864f762c45c43cee6"} Nov 04 04:51:51.095660 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:51.095594 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-5gpsn" Nov 04 04:51:51.202366 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:51.202321 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9cxmr" podStartSLOduration=1.598176863 podStartE2EDuration="6.202291929s" totalImagesPullingTime="4.604115066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:51:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:51:51.20212718 +0000 UTC m=+352.892209626" watchObservedRunningTime="2026-11-04 04:51:51.202291929 +0000 UTC m=+352.892374377" Nov 04 04:51:51.311958 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:51.311915 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-5gpsn" podStartSLOduration=2.090634459 podStartE2EDuration="6.311899758s" totalImagesPullingTime="4.221265299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:51:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:51:51.310643326 +0000 UTC m=+353.000725816" watchObservedRunningTime="2026-11-04 04:51:51.311899758 +0000 UTC m=+353.001982205" Nov 04 04:51:55.060506 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:51:55.060477 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6c844db6b-c5mnw" Nov 04 04:52:02.108793 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:02.108759 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-5gpsn" Nov 04 04:52:03.455770 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:52:03.455734 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 04:52:22.108904 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:22.108874 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9cxmr" Nov 04 04:52:24.531487 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:24.531456 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-86c9fb87b5-7nbc6"] Nov 04 04:52:24.538377 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:24.538353 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-86c9fb87b5-7nbc6" Nov 04 04:52:24.633212 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:24.633183 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 04 04:52:24.633430 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:24.633411 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 04 04:52:24.633517 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:24.633489 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-mgtmp\"" Nov 04 04:52:24.635164 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:24.635144 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-86c9fb87b5-7nbc6"] Nov 04 04:52:24.653113 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:24.653090 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e0fa121-36d5-4003-ac66-244badf594c3-cert\") pod \"jobset-controller-manager-86c9fb87b5-7nbc6\" (UID: \"0e0fa121-36d5-4003-ac66-244badf594c3\") " pod="openshift-jobset-operator/jobset-controller-manager-86c9fb87b5-7nbc6" Nov 04 04:52:24.653209 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:24.653128 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0e0fa121-36d5-4003-ac66-244badf594c3-manager-config\") pod \"jobset-controller-manager-86c9fb87b5-7nbc6\" (UID: \"0e0fa121-36d5-4003-ac66-244badf594c3\") " pod="openshift-jobset-operator/jobset-controller-manager-86c9fb87b5-7nbc6" Nov 04 04:52:24.653209 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:24.653161 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e0fa121-36d5-4003-ac66-244badf594c3-metrics-certs\") pod \"jobset-controller-manager-86c9fb87b5-7nbc6\" (UID: \"0e0fa121-36d5-4003-ac66-244badf594c3\") " pod="openshift-jobset-operator/jobset-controller-manager-86c9fb87b5-7nbc6" Nov 04 04:52:24.653282 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:24.653202 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl4g6\" (UniqueName: \"kubernetes.io/projected/0e0fa121-36d5-4003-ac66-244badf594c3-kube-api-access-nl4g6\") pod \"jobset-controller-manager-86c9fb87b5-7nbc6\" (UID: \"0e0fa121-36d5-4003-ac66-244badf594c3\") " pod="openshift-jobset-operator/jobset-controller-manager-86c9fb87b5-7nbc6" Nov 04 04:52:24.679638 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:24.679613 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 04 04:52:24.753979 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:24.753952 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e0fa121-36d5-4003-ac66-244badf594c3-metrics-certs\") pod \"jobset-controller-manager-86c9fb87b5-7nbc6\" (UID: \"0e0fa121-36d5-4003-ac66-244badf594c3\") " pod="openshift-jobset-operator/jobset-controller-manager-86c9fb87b5-7nbc6" Nov 04 04:52:24.754080 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:24.753996 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl4g6\" (UniqueName: \"kubernetes.io/projected/0e0fa121-36d5-4003-ac66-244badf594c3-kube-api-access-nl4g6\") pod \"jobset-controller-manager-86c9fb87b5-7nbc6\" (UID: \"0e0fa121-36d5-4003-ac66-244badf594c3\") " pod="openshift-jobset-operator/jobset-controller-manager-86c9fb87b5-7nbc6" Nov 04 04:52:24.754080 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:24.754047 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e0fa121-36d5-4003-ac66-244badf594c3-cert\") pod \"jobset-controller-manager-86c9fb87b5-7nbc6\" (UID: \"0e0fa121-36d5-4003-ac66-244badf594c3\") " pod="openshift-jobset-operator/jobset-controller-manager-86c9fb87b5-7nbc6" Nov 04 04:52:24.754198 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:24.754095 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0e0fa121-36d5-4003-ac66-244badf594c3-manager-config\") pod \"jobset-controller-manager-86c9fb87b5-7nbc6\" (UID: \"0e0fa121-36d5-4003-ac66-244badf594c3\") " pod="openshift-jobset-operator/jobset-controller-manager-86c9fb87b5-7nbc6" Nov 04 04:52:24.754801 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:24.754779 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0e0fa121-36d5-4003-ac66-244badf594c3-manager-config\") pod \"jobset-controller-manager-86c9fb87b5-7nbc6\" (UID: \"0e0fa121-36d5-4003-ac66-244badf594c3\") " pod="openshift-jobset-operator/jobset-controller-manager-86c9fb87b5-7nbc6" Nov 04 04:52:24.756427 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:24.756405 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e0fa121-36d5-4003-ac66-244badf594c3-cert\") pod \"jobset-controller-manager-86c9fb87b5-7nbc6\" (UID: \"0e0fa121-36d5-4003-ac66-244badf594c3\") " pod="openshift-jobset-operator/jobset-controller-manager-86c9fb87b5-7nbc6" Nov 04 04:52:24.756509 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:24.756409 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e0fa121-36d5-4003-ac66-244badf594c3-metrics-certs\") pod \"jobset-controller-manager-86c9fb87b5-7nbc6\" (UID: \"0e0fa121-36d5-4003-ac66-244badf594c3\") " pod="openshift-jobset-operator/jobset-controller-manager-86c9fb87b5-7nbc6" Nov 04 04:52:24.811196 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:24.811173 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl4g6\" (UniqueName: \"kubernetes.io/projected/0e0fa121-36d5-4003-ac66-244badf594c3-kube-api-access-nl4g6\") pod \"jobset-controller-manager-86c9fb87b5-7nbc6\" (UID: \"0e0fa121-36d5-4003-ac66-244badf594c3\") " pod="openshift-jobset-operator/jobset-controller-manager-86c9fb87b5-7nbc6" Nov 04 04:52:24.849130 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:24.849098 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-86c9fb87b5-7nbc6" Nov 04 04:52:25.220274 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:25.220251 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-86c9fb87b5-7nbc6"] Nov 04 04:52:25.222375 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:52:25.222343 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e0fa121_36d5_4003_ac66_244badf594c3.slice/crio-baaf7e8c992e9efbd63f9fa824b5805959cdf0134b196b5fb95bf81bfc35725f WatchSource:0}: Error finding container baaf7e8c992e9efbd63f9fa824b5805959cdf0134b196b5fb95bf81bfc35725f: Status 404 returned error can't find the container with id baaf7e8c992e9efbd63f9fa824b5805959cdf0134b196b5fb95bf81bfc35725f Nov 04 04:52:25.262385 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:25.262355 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-86c9fb87b5-7nbc6" event={"ID":"0e0fa121-36d5-4003-ac66-244badf594c3","Type":"ContainerStarted","Data":"baaf7e8c992e9efbd63f9fa824b5805959cdf0134b196b5fb95bf81bfc35725f"} Nov 04 04:52:31.311510 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:31.311457 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-86c9fb87b5-7nbc6" event={"ID":"0e0fa121-36d5-4003-ac66-244badf594c3","Type":"ContainerStarted","Data":"837a7ed6b30b2499fd852bd0ce037f404ca13cf0aaf7addf7dd1f8256e366792"} Nov 04 04:52:31.311925 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:31.311557 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-86c9fb87b5-7nbc6" Nov 04 04:52:31.422336 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:31.422241 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-86c9fb87b5-7nbc6" podStartSLOduration=1.518131904 podStartE2EDuration="7.422229451s" totalImagesPullingTime="5.904097547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:52:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:52:31.421820593 +0000 UTC m=+393.111903042" watchObservedRunningTime="2026-11-04 04:52:31.422229451 +0000 UTC m=+393.112311899" Nov 04 04:52:33.456532 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:52:33.456496 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 04:52:34.328012 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:34.327984 4395 generic.go:356] "Generic (PLEG): container finished" podID="115eb9f0-c97e-4133-939f-1797a4157820" containerID="5346ce7606aa23646860e1c24f3939bf7231b3d84c1c22054764b003337b0440" exitCode=0 Nov 04 04:52:34.328170 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:34.328051 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" event={"ID":"115eb9f0-c97e-4133-939f-1797a4157820","Type":"ContainerDied","Data":"5346ce7606aa23646860e1c24f3939bf7231b3d84c1c22054764b003337b0440"} Nov 04 04:52:34.328406 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:34.328390 4395 scope.go:122] "RemoveContainer" containerID="5346ce7606aa23646860e1c24f3939bf7231b3d84c1c22054764b003337b0440" Nov 04 04:52:34.329407 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:34.329376 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 04:52:34.329489 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:34.329455 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 04:52:34.329529 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:34.329492 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 04:52:35.334937 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:35.334906 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" event={"ID":"115eb9f0-c97e-4133-939f-1797a4157820","Type":"ContainerStarted","Data":"b637ab3eec393845c57f37ef4a549aa08f5ce88f4b775f2ac809b7f2703786ec"} Nov 04 04:52:36.075321 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:36.071660 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" Nov 04 04:52:36.075321 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:36.071784 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-qrtfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body= Nov 04 04:52:36.075321 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:36.071830 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" podUID="115eb9f0-c97e-4133-939f-1797a4157820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" Nov 04 04:52:37.071745 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:37.071705 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-qrtfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body= Nov 04 04:52:37.072424 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:37.072381 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" podUID="115eb9f0-c97e-4133-939f-1797a4157820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" Nov 04 04:52:38.068322 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:38.065723 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-qrtfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body= Nov 04 04:52:38.068322 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:38.065795 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" podUID="115eb9f0-c97e-4133-939f-1797a4157820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" Nov 04 04:52:39.065375 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:39.061669 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-qrtfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body= Nov 04 04:52:39.065375 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:39.061729 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" podUID="115eb9f0-c97e-4133-939f-1797a4157820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" Nov 04 04:52:40.068682 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:40.068643 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-qrtfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body= Nov 04 04:52:40.069247 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:40.069215 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" podUID="115eb9f0-c97e-4133-939f-1797a4157820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" Nov 04 04:52:41.063978 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:41.063950 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" Nov 04 04:52:41.064253 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:41.064219 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" Nov 04 04:52:41.065007 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:41.064990 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" Nov 04 04:52:42.322419 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:52:42.322388 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-86c9fb87b5-7nbc6" Nov 04 04:53:03.457574 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:53:03.457533 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 04:53:33.457635 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:53:33.457595 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 04:53:35.059894 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:53:35.059860 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-9cxmr"] Nov 04 04:53:35.060320 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:53:35.060076 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9cxmr" podUID="a3759482-cd05-4daf-ad79-93a2e9e1e35b" containerName="manager" containerID="cri-o://2ec2ec37f907212571dec987531f3fd0100828cc81cd5c18b70e7d5c889cce0b" gracePeriod=30 Nov 04 04:53:35.298756 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:53:35.298734 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9cxmr" Nov 04 04:53:35.355426 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:53:35.355379 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3759482-cd05-4daf-ad79-93a2e9e1e35b-cert\" (UniqueName: \"kubernetes.io/secret/a3759482-cd05-4daf-ad79-93a2e9e1e35b-cert\") pod \"a3759482-cd05-4daf-ad79-93a2e9e1e35b\" (UID: \"a3759482-cd05-4daf-ad79-93a2e9e1e35b\") " Nov 04 04:53:35.355426 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:53:35.355425 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3759482-cd05-4daf-ad79-93a2e9e1e35b-kube-api-access-gq74k\" (UniqueName: \"kubernetes.io/projected/a3759482-cd05-4daf-ad79-93a2e9e1e35b-kube-api-access-gq74k\") pod \"a3759482-cd05-4daf-ad79-93a2e9e1e35b\" (UID: \"a3759482-cd05-4daf-ad79-93a2e9e1e35b\") " Nov 04 04:53:35.357446 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:53:35.357418 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3759482-cd05-4daf-ad79-93a2e9e1e35b-cert" pod "a3759482-cd05-4daf-ad79-93a2e9e1e35b" (UID: "a3759482-cd05-4daf-ad79-93a2e9e1e35b"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 04:53:35.357532 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:53:35.357519 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3759482-cd05-4daf-ad79-93a2e9e1e35b-kube-api-access-gq74k" pod "a3759482-cd05-4daf-ad79-93a2e9e1e35b" (UID: "a3759482-cd05-4daf-ad79-93a2e9e1e35b"). InnerVolumeSpecName "kube-api-access-gq74k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 04:53:35.456784 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:53:35.456759 4395 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3759482-cd05-4daf-ad79-93a2e9e1e35b-cert\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:53:35.456784 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:53:35.456786 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq74k\" (UniqueName: \"kubernetes.io/projected/a3759482-cd05-4daf-ad79-93a2e9e1e35b-kube-api-access-gq74k\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 04:53:35.641051 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:53:35.640980 4395 generic.go:356] "Generic (PLEG): container finished" podID="a3759482-cd05-4daf-ad79-93a2e9e1e35b" containerID="2ec2ec37f907212571dec987531f3fd0100828cc81cd5c18b70e7d5c889cce0b" exitCode=0 Nov 04 04:53:35.641051 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:53:35.641020 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9cxmr" Nov 04 04:53:35.641051 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:53:35.641037 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9cxmr" event={"ID":"a3759482-cd05-4daf-ad79-93a2e9e1e35b","Type":"ContainerDied","Data":"2ec2ec37f907212571dec987531f3fd0100828cc81cd5c18b70e7d5c889cce0b"} Nov 04 04:53:35.641242 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:53:35.641077 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9cxmr" event={"ID":"a3759482-cd05-4daf-ad79-93a2e9e1e35b","Type":"ContainerDied","Data":"866983afa324bf680c558dbec37ba56d51dbe89d72d0a8a900215bdfbd3790e5"} Nov 04 04:53:35.641242 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:53:35.641098 4395 scope.go:122] "RemoveContainer" containerID="2ec2ec37f907212571dec987531f3fd0100828cc81cd5c18b70e7d5c889cce0b" Nov 04 04:53:35.649740 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:53:35.649722 4395 scope.go:122] "RemoveContainer" containerID="2ec2ec37f907212571dec987531f3fd0100828cc81cd5c18b70e7d5c889cce0b" Nov 04 04:53:35.649999 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:53:35.649981 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ec2ec37f907212571dec987531f3fd0100828cc81cd5c18b70e7d5c889cce0b\": container with ID starting with 2ec2ec37f907212571dec987531f3fd0100828cc81cd5c18b70e7d5c889cce0b not found: ID does not exist" containerID="2ec2ec37f907212571dec987531f3fd0100828cc81cd5c18b70e7d5c889cce0b" Nov 04 04:53:35.650051 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:53:35.650008 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ec2ec37f907212571dec987531f3fd0100828cc81cd5c18b70e7d5c889cce0b"} err="failed to get container status \"2ec2ec37f907212571dec987531f3fd0100828cc81cd5c18b70e7d5c889cce0b\": rpc error: code = NotFound desc = could not find container \"2ec2ec37f907212571dec987531f3fd0100828cc81cd5c18b70e7d5c889cce0b\": container with ID starting with 2ec2ec37f907212571dec987531f3fd0100828cc81cd5c18b70e7d5c889cce0b not found: ID does not exist" Nov 04 04:53:35.795833 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:53:35.795803 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-9cxmr"] Nov 04 04:53:35.840451 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:53:35.840425 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-9cxmr"] Nov 04 04:53:36.414673 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:53:36.414643 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3759482-cd05-4daf-ad79-93a2e9e1e35b" path="/var/lib/kubelet/pods/a3759482-cd05-4daf-ad79-93a2e9e1e35b/volumes" Nov 04 04:54:02.539129 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:02.539101 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-ks5ds"] Nov 04 04:54:02.539562 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:02.539428 4395 cpu_manager.go:414] "containerMap: removing container" podUID="a3759482-cd05-4daf-ad79-93a2e9e1e35b" containerName="manager" Nov 04 04:54:02.539562 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:02.539439 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3759482-cd05-4daf-ad79-93a2e9e1e35b" containerName="manager" Nov 04 04:54:02.539562 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:02.539509 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3759482-cd05-4daf-ad79-93a2e9e1e35b" containerName="manager" Nov 04 04:54:02.543732 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:02.543712 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-ks5ds" Nov 04 04:54:02.592530 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:02.592492 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 04 04:54:02.592638 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:02.592570 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 04 04:54:02.638674 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:02.638647 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-67mz4\"" Nov 04 04:54:02.638806 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:02.638785 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 04 04:54:02.640469 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:02.640445 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-ks5ds"] Nov 04 04:54:02.660100 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:02.660072 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bss9\" (UniqueName: \"kubernetes.io/projected/1fbe78fb-a1b4-4faa-8f86-e7df56474b9a-kube-api-access-9bss9\") pod \"echo-simulator-kserve-64b6f8f96c-ks5ds\" (UID: \"1fbe78fb-a1b4-4faa-8f86-e7df56474b9a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ks5ds" Nov 04 04:54:02.660215 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:02.660130 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fbe78fb-a1b4-4faa-8f86-e7df56474b9a-home\") pod \"echo-simulator-kserve-64b6f8f96c-ks5ds\" (UID: \"1fbe78fb-a1b4-4faa-8f86-e7df56474b9a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ks5ds" Nov 04 04:54:02.660287 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:02.660218 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fbe78fb-a1b4-4faa-8f86-e7df56474b9a-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-ks5ds\" (UID: \"1fbe78fb-a1b4-4faa-8f86-e7df56474b9a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ks5ds" Nov 04 04:54:02.660382 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:02.660281 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fbe78fb-a1b4-4faa-8f86-e7df56474b9a-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-ks5ds\" (UID: \"1fbe78fb-a1b4-4faa-8f86-e7df56474b9a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ks5ds" Nov 04 04:54:02.660382 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:02.660324 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fbe78fb-a1b4-4faa-8f86-e7df56474b9a-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-ks5ds\" (UID: \"1fbe78fb-a1b4-4faa-8f86-e7df56474b9a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ks5ds" Nov 04 04:54:02.660382 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:02.660376 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fbe78fb-a1b4-4faa-8f86-e7df56474b9a-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-ks5ds\" (UID: \"1fbe78fb-a1b4-4faa-8f86-e7df56474b9a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ks5ds" Nov 04 04:54:02.761269 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:02.761240 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bss9\" (UniqueName: \"kubernetes.io/projected/1fbe78fb-a1b4-4faa-8f86-e7df56474b9a-kube-api-access-9bss9\") pod \"echo-simulator-kserve-64b6f8f96c-ks5ds\" (UID: \"1fbe78fb-a1b4-4faa-8f86-e7df56474b9a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ks5ds" Nov 04 04:54:02.761396 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:02.761293 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fbe78fb-a1b4-4faa-8f86-e7df56474b9a-home\") pod \"echo-simulator-kserve-64b6f8f96c-ks5ds\" (UID: \"1fbe78fb-a1b4-4faa-8f86-e7df56474b9a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ks5ds" Nov 04 04:54:02.761396 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:02.761342 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fbe78fb-a1b4-4faa-8f86-e7df56474b9a-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-ks5ds\" (UID: \"1fbe78fb-a1b4-4faa-8f86-e7df56474b9a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ks5ds" Nov 04 04:54:02.761396 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:02.761360 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fbe78fb-a1b4-4faa-8f86-e7df56474b9a-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-ks5ds\" (UID: \"1fbe78fb-a1b4-4faa-8f86-e7df56474b9a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ks5ds" Nov 04 04:54:02.761396 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:02.761375 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fbe78fb-a1b4-4faa-8f86-e7df56474b9a-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-ks5ds\" (UID: \"1fbe78fb-a1b4-4faa-8f86-e7df56474b9a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ks5ds" Nov 04 04:54:02.761615 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:02.761407 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fbe78fb-a1b4-4faa-8f86-e7df56474b9a-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-ks5ds\" (UID: \"1fbe78fb-a1b4-4faa-8f86-e7df56474b9a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ks5ds" Nov 04 04:54:02.761845 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:02.761798 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fbe78fb-a1b4-4faa-8f86-e7df56474b9a-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-ks5ds\" (UID: \"1fbe78fb-a1b4-4faa-8f86-e7df56474b9a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ks5ds" Nov 04 04:54:02.761936 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:02.761846 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fbe78fb-a1b4-4faa-8f86-e7df56474b9a-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-ks5ds\" (UID: \"1fbe78fb-a1b4-4faa-8f86-e7df56474b9a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ks5ds" Nov 04 04:54:02.761936 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:02.761901 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fbe78fb-a1b4-4faa-8f86-e7df56474b9a-home\") pod \"echo-simulator-kserve-64b6f8f96c-ks5ds\" (UID: \"1fbe78fb-a1b4-4faa-8f86-e7df56474b9a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ks5ds" Nov 04 04:54:02.763922 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:02.763898 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fbe78fb-a1b4-4faa-8f86-e7df56474b9a-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-ks5ds\" (UID: \"1fbe78fb-a1b4-4faa-8f86-e7df56474b9a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ks5ds" Nov 04 04:54:02.764136 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:02.764121 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fbe78fb-a1b4-4faa-8f86-e7df56474b9a-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-ks5ds\" (UID: \"1fbe78fb-a1b4-4faa-8f86-e7df56474b9a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ks5ds" Nov 04 04:54:02.819479 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:02.819452 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bss9\" (UniqueName: \"kubernetes.io/projected/1fbe78fb-a1b4-4faa-8f86-e7df56474b9a-kube-api-access-9bss9\") pod \"echo-simulator-kserve-64b6f8f96c-ks5ds\" (UID: \"1fbe78fb-a1b4-4faa-8f86-e7df56474b9a\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ks5ds" Nov 04 04:54:02.856504 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:02.856481 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-ks5ds" Nov 04 04:54:03.229622 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:03.229595 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-ks5ds"] Nov 04 04:54:03.232528 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:54:03.232500 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fbe78fb_a1b4_4faa_8f86_e7df56474b9a.slice/crio-b62c3e9b3170d744983c73e2ed8343b89cf63e5931db8dd06fa2a98ad2f1cc5d WatchSource:0}: Error finding container b62c3e9b3170d744983c73e2ed8343b89cf63e5931db8dd06fa2a98ad2f1cc5d: Status 404 returned error can't find the container with id b62c3e9b3170d744983c73e2ed8343b89cf63e5931db8dd06fa2a98ad2f1cc5d Nov 04 04:54:03.456409 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:54:03.456381 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 04:54:03.774791 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:03.774760 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-ks5ds" event={"ID":"1fbe78fb-a1b4-4faa-8f86-e7df56474b9a","Type":"ContainerStarted","Data":"b62c3e9b3170d744983c73e2ed8343b89cf63e5931db8dd06fa2a98ad2f1cc5d"} Nov 04 04:54:05.788827 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:05.788800 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-ks5ds" event={"ID":"1fbe78fb-a1b4-4faa-8f86-e7df56474b9a","Type":"ContainerStarted","Data":"3c38ef1164224d6a1d55f63eab0fa09a55b63cc44b079ca9a2d2709db729f4d1"} Nov 04 04:54:05.857163 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:05.857132 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-ks5ds" Nov 04 04:54:05.900519 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:05.898788 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-ks5ds" podStartSLOduration=1.4822590789999999 podStartE2EDuration="3.898770455s" totalImagesPullingTime="2.416511376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:54:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:54:05.897846399 +0000 UTC m=+487.587928846" watchObservedRunningTime="2026-11-04 04:54:05.898770455 +0000 UTC m=+487.588852903" Nov 04 04:54:12.857607 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:12.857533 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-ks5ds" Nov 04 04:54:12.870030 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:12.870007 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-ks5ds" Nov 04 04:54:13.038616 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:13.038595 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs"] Nov 04 04:54:13.042108 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:13.042091 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs" Nov 04 04:54:13.091049 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:13.091030 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 04 04:54:13.105530 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:13.105504 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs"] Nov 04 04:54:13.140981 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:13.140928 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfa8140b-9e2c-418e-bee1-6394bc7a6ac7-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs\" (UID: \"cfa8140b-9e2c-418e-bee1-6394bc7a6ac7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs" Nov 04 04:54:13.140981 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:13.140957 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfa8140b-9e2c-418e-bee1-6394bc7a6ac7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs\" (UID: \"cfa8140b-9e2c-418e-bee1-6394bc7a6ac7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs" Nov 04 04:54:13.140981 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:13.140976 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfa8140b-9e2c-418e-bee1-6394bc7a6ac7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs\" (UID: \"cfa8140b-9e2c-418e-bee1-6394bc7a6ac7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs" Nov 04 04:54:13.141141 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:13.141037 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfa8140b-9e2c-418e-bee1-6394bc7a6ac7-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs\" (UID: \"cfa8140b-9e2c-418e-bee1-6394bc7a6ac7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs" Nov 04 04:54:13.141141 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:13.141129 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdmcx\" (UniqueName: \"kubernetes.io/projected/cfa8140b-9e2c-418e-bee1-6394bc7a6ac7-kube-api-access-xdmcx\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs\" (UID: \"cfa8140b-9e2c-418e-bee1-6394bc7a6ac7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs" Nov 04 04:54:13.141204 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:13.141173 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfa8140b-9e2c-418e-bee1-6394bc7a6ac7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs\" (UID: \"cfa8140b-9e2c-418e-bee1-6394bc7a6ac7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs" Nov 04 04:54:13.241994 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:13.241961 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfa8140b-9e2c-418e-bee1-6394bc7a6ac7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs\" (UID: \"cfa8140b-9e2c-418e-bee1-6394bc7a6ac7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs" Nov 04 04:54:13.242163 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:13.242015 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfa8140b-9e2c-418e-bee1-6394bc7a6ac7-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs\" (UID: \"cfa8140b-9e2c-418e-bee1-6394bc7a6ac7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs" Nov 04 04:54:13.242163 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:13.242030 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfa8140b-9e2c-418e-bee1-6394bc7a6ac7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs\" (UID: \"cfa8140b-9e2c-418e-bee1-6394bc7a6ac7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs" Nov 04 04:54:13.242163 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:13.242045 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfa8140b-9e2c-418e-bee1-6394bc7a6ac7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs\" (UID: \"cfa8140b-9e2c-418e-bee1-6394bc7a6ac7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs" Nov 04 04:54:13.242163 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:13.242074 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfa8140b-9e2c-418e-bee1-6394bc7a6ac7-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs\" (UID: \"cfa8140b-9e2c-418e-bee1-6394bc7a6ac7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs" Nov 04 04:54:13.242163 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:13.242106 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdmcx\" (UniqueName: \"kubernetes.io/projected/cfa8140b-9e2c-418e-bee1-6394bc7a6ac7-kube-api-access-xdmcx\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs\" (UID: \"cfa8140b-9e2c-418e-bee1-6394bc7a6ac7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs" Nov 04 04:54:13.242527 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:13.242495 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfa8140b-9e2c-418e-bee1-6394bc7a6ac7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs\" (UID: \"cfa8140b-9e2c-418e-bee1-6394bc7a6ac7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs" Nov 04 04:54:13.242527 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:13.242517 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfa8140b-9e2c-418e-bee1-6394bc7a6ac7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs\" (UID: \"cfa8140b-9e2c-418e-bee1-6394bc7a6ac7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs" Nov 04 04:54:13.242608 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:13.242563 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfa8140b-9e2c-418e-bee1-6394bc7a6ac7-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs\" (UID: \"cfa8140b-9e2c-418e-bee1-6394bc7a6ac7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs" Nov 04 04:54:13.244317 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:13.244266 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfa8140b-9e2c-418e-bee1-6394bc7a6ac7-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs\" (UID: \"cfa8140b-9e2c-418e-bee1-6394bc7a6ac7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs" Nov 04 04:54:13.244668 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:13.244648 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfa8140b-9e2c-418e-bee1-6394bc7a6ac7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs\" (UID: \"cfa8140b-9e2c-418e-bee1-6394bc7a6ac7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs" Nov 04 04:54:13.298923 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:13.298896 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdmcx\" (UniqueName: \"kubernetes.io/projected/cfa8140b-9e2c-418e-bee1-6394bc7a6ac7-kube-api-access-xdmcx\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs\" (UID: \"cfa8140b-9e2c-418e-bee1-6394bc7a6ac7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs" Nov 04 04:54:13.354192 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:13.354169 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs" Nov 04 04:54:13.728466 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:13.728442 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs"] Nov 04 04:54:13.730634 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:54:13.730606 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa8140b_9e2c_418e_bee1_6394bc7a6ac7.slice/crio-1befba7caa809d37edcbf19e67ef848fcb26ea6722480143d2e22d275359b7e3 WatchSource:0}: Error finding container 1befba7caa809d37edcbf19e67ef848fcb26ea6722480143d2e22d275359b7e3: Status 404 returned error can't find the container with id 1befba7caa809d37edcbf19e67ef848fcb26ea6722480143d2e22d275359b7e3 Nov 04 04:54:13.827720 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:13.827689 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs" event={"ID":"cfa8140b-9e2c-418e-bee1-6394bc7a6ac7","Type":"ContainerStarted","Data":"1befba7caa809d37edcbf19e67ef848fcb26ea6722480143d2e22d275359b7e3"} Nov 04 04:54:13.838857 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:13.838839 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-ks5ds" Nov 04 04:54:15.840676 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:15.840645 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs" event={"ID":"cfa8140b-9e2c-418e-bee1-6394bc7a6ac7","Type":"ContainerStarted","Data":"e142e7c64909584d06ed7067d7ec73bb57b9cbe0d014580559823adabc23677d"} Nov 04 04:54:15.954984 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:15.954938 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs" podStartSLOduration=2.443775358 podStartE2EDuration="3.954924407s" totalImagesPullingTime="1.511149049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:54:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:54:15.953941591 +0000 UTC m=+497.644024039" watchObservedRunningTime="2026-11-04 04:54:15.954924407 +0000 UTC m=+497.645006855" Nov 04 04:54:16.355167 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:16.355139 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs" Nov 04 04:54:18.094728 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:18.094693 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-rzts7"] Nov 04 04:54:18.098445 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:18.098422 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rzts7" Nov 04 04:54:18.182467 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:18.182440 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/102341e6-1ba0-40cd-b185-9f79f0310814-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rzts7\" (UID: \"102341e6-1ba0-40cd-b185-9f79f0310814\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rzts7" Nov 04 04:54:18.182599 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:18.182476 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/102341e6-1ba0-40cd-b185-9f79f0310814-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rzts7\" (UID: \"102341e6-1ba0-40cd-b185-9f79f0310814\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rzts7" Nov 04 04:54:18.182643 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:18.182625 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/102341e6-1ba0-40cd-b185-9f79f0310814-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rzts7\" (UID: \"102341e6-1ba0-40cd-b185-9f79f0310814\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rzts7" Nov 04 04:54:18.182684 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:18.182650 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwwv8\" (UniqueName: \"kubernetes.io/projected/102341e6-1ba0-40cd-b185-9f79f0310814-kube-api-access-gwwv8\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rzts7\" (UID: \"102341e6-1ba0-40cd-b185-9f79f0310814\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rzts7" Nov 04 04:54:18.182684 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:18.182675 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/102341e6-1ba0-40cd-b185-9f79f0310814-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rzts7\" (UID: \"102341e6-1ba0-40cd-b185-9f79f0310814\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rzts7" Nov 04 04:54:18.182749 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:18.182691 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/102341e6-1ba0-40cd-b185-9f79f0310814-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rzts7\" (UID: \"102341e6-1ba0-40cd-b185-9f79f0310814\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rzts7" Nov 04 04:54:18.193367 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:18.193346 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 04 04:54:18.195074 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:18.195054 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-rzts7"] Nov 04 04:54:18.283543 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:18.283515 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/102341e6-1ba0-40cd-b185-9f79f0310814-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rzts7\" (UID: \"102341e6-1ba0-40cd-b185-9f79f0310814\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rzts7" Nov 04 04:54:18.283671 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:18.283550 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/102341e6-1ba0-40cd-b185-9f79f0310814-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rzts7\" (UID: \"102341e6-1ba0-40cd-b185-9f79f0310814\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rzts7" Nov 04 04:54:18.283671 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:18.283608 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/102341e6-1ba0-40cd-b185-9f79f0310814-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rzts7\" (UID: \"102341e6-1ba0-40cd-b185-9f79f0310814\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rzts7" Nov 04 04:54:18.283671 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:18.283625 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwwv8\" (UniqueName: \"kubernetes.io/projected/102341e6-1ba0-40cd-b185-9f79f0310814-kube-api-access-gwwv8\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rzts7\" (UID: \"102341e6-1ba0-40cd-b185-9f79f0310814\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rzts7" Nov 04 04:54:18.283671 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:18.283659 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/102341e6-1ba0-40cd-b185-9f79f0310814-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rzts7\" (UID: \"102341e6-1ba0-40cd-b185-9f79f0310814\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rzts7" Nov 04 04:54:18.283919 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:18.283896 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/102341e6-1ba0-40cd-b185-9f79f0310814-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rzts7\" (UID: \"102341e6-1ba0-40cd-b185-9f79f0310814\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rzts7" Nov 04 04:54:18.284011 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:18.283988 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/102341e6-1ba0-40cd-b185-9f79f0310814-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rzts7\" (UID: \"102341e6-1ba0-40cd-b185-9f79f0310814\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rzts7" Nov 04 04:54:18.284060 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:18.284000 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/102341e6-1ba0-40cd-b185-9f79f0310814-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rzts7\" (UID: \"102341e6-1ba0-40cd-b185-9f79f0310814\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rzts7" Nov 04 04:54:18.284239 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:18.284221 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/102341e6-1ba0-40cd-b185-9f79f0310814-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rzts7\" (UID: \"102341e6-1ba0-40cd-b185-9f79f0310814\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rzts7" Nov 04 04:54:18.285913 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:18.285895 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/102341e6-1ba0-40cd-b185-9f79f0310814-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rzts7\" (UID: \"102341e6-1ba0-40cd-b185-9f79f0310814\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rzts7" Nov 04 04:54:18.286175 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:18.286159 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/102341e6-1ba0-40cd-b185-9f79f0310814-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rzts7\" (UID: \"102341e6-1ba0-40cd-b185-9f79f0310814\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rzts7" Nov 04 04:54:18.341018 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:18.340997 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwwv8\" (UniqueName: \"kubernetes.io/projected/102341e6-1ba0-40cd-b185-9f79f0310814-kube-api-access-gwwv8\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-rzts7\" (UID: \"102341e6-1ba0-40cd-b185-9f79f0310814\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rzts7" Nov 04 04:54:18.410976 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:18.410911 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rzts7" Nov 04 04:54:18.783498 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:18.783475 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-rzts7"] Nov 04 04:54:18.787114 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:54:18.787086 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod102341e6_1ba0_40cd_b185_9f79f0310814.slice/crio-6d54d3f7ac948cfa562f070d57cd41fab69bf98c46ff82f33941ee002f6643b9 WatchSource:0}: Error finding container 6d54d3f7ac948cfa562f070d57cd41fab69bf98c46ff82f33941ee002f6643b9: Status 404 returned error can't find the container with id 6d54d3f7ac948cfa562f070d57cd41fab69bf98c46ff82f33941ee002f6643b9 Nov 04 04:54:18.855794 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:18.855768 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rzts7" event={"ID":"102341e6-1ba0-40cd-b185-9f79f0310814","Type":"ContainerStarted","Data":"6d54d3f7ac948cfa562f070d57cd41fab69bf98c46ff82f33941ee002f6643b9"} Nov 04 04:54:19.861803 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:19.861769 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rzts7" event={"ID":"102341e6-1ba0-40cd-b185-9f79f0310814","Type":"ContainerStarted","Data":"3c3b42b35fbe5169fa6af0faab3369485f8ca8b56d70ddb59dc3c27f10507a93"} Nov 04 04:54:20.414901 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:20.414866 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rzts7" Nov 04 04:54:23.354778 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:23.354740 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs" Nov 04 04:54:23.367015 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:23.366987 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs" Nov 04 04:54:23.477455 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:23.477412 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rzts7" podStartSLOduration=4.870166678 podStartE2EDuration="5.477399667s" totalImagesPullingTime="607.232989ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:54:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:54:19.971217245 +0000 UTC m=+501.661299692" watchObservedRunningTime="2026-11-04 04:54:23.477399667 +0000 UTC m=+505.167482116" Nov 04 04:54:23.891734 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:23.891711 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs" Nov 04 04:54:28.415153 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:28.415119 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rzts7" Nov 04 04:54:28.426333 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:28.426287 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rzts7" Nov 04 04:54:28.917663 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:28.917637 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-rzts7" Nov 04 04:54:33.457095 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:54:33.457059 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 04:54:36.176250 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:36.176219 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z"] Nov 04 04:54:36.179867 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:36.179847 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" Nov 04 04:54:36.278713 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:36.278687 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z"] Nov 04 04:54:36.331944 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:36.331915 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b948fdc0-b375-455d-8669-d47c9d84e8de-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7bc9z\" (UID: \"b948fdc0-b375-455d-8669-d47c9d84e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" Nov 04 04:54:36.332066 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:36.331968 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b948fdc0-b375-455d-8669-d47c9d84e8de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7bc9z\" (UID: \"b948fdc0-b375-455d-8669-d47c9d84e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" Nov 04 04:54:36.332066 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:36.332023 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpwdr\" (UniqueName: \"kubernetes.io/projected/b948fdc0-b375-455d-8669-d47c9d84e8de-kube-api-access-jpwdr\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7bc9z\" (UID: \"b948fdc0-b375-455d-8669-d47c9d84e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" Nov 04 04:54:36.332066 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:36.332053 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b948fdc0-b375-455d-8669-d47c9d84e8de-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7bc9z\" (UID: \"b948fdc0-b375-455d-8669-d47c9d84e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" Nov 04 04:54:36.332197 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:36.332075 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b948fdc0-b375-455d-8669-d47c9d84e8de-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7bc9z\" (UID: \"b948fdc0-b375-455d-8669-d47c9d84e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" Nov 04 04:54:36.332197 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:36.332111 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b948fdc0-b375-455d-8669-d47c9d84e8de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7bc9z\" (UID: \"b948fdc0-b375-455d-8669-d47c9d84e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" Nov 04 04:54:36.332197 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:36.332171 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b948fdc0-b375-455d-8669-d47c9d84e8de-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7bc9z\" (UID: \"b948fdc0-b375-455d-8669-d47c9d84e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" Nov 04 04:54:36.332197 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:36.332188 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b948fdc0-b375-455d-8669-d47c9d84e8de-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7bc9z\" (UID: \"b948fdc0-b375-455d-8669-d47c9d84e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" Nov 04 04:54:36.332405 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:36.332221 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b948fdc0-b375-455d-8669-d47c9d84e8de-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7bc9z\" (UID: \"b948fdc0-b375-455d-8669-d47c9d84e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" Nov 04 04:54:36.432712 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:36.432643 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b948fdc0-b375-455d-8669-d47c9d84e8de-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7bc9z\" (UID: \"b948fdc0-b375-455d-8669-d47c9d84e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" Nov 04 04:54:36.432712 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:36.432705 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b948fdc0-b375-455d-8669-d47c9d84e8de-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7bc9z\" (UID: \"b948fdc0-b375-455d-8669-d47c9d84e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" Nov 04 04:54:36.432908 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:36.432757 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b948fdc0-b375-455d-8669-d47c9d84e8de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7bc9z\" (UID: \"b948fdc0-b375-455d-8669-d47c9d84e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" Nov 04 04:54:36.432908 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:36.432784 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpwdr\" (UniqueName: \"kubernetes.io/projected/b948fdc0-b375-455d-8669-d47c9d84e8de-kube-api-access-jpwdr\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7bc9z\" (UID: \"b948fdc0-b375-455d-8669-d47c9d84e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" Nov 04 04:54:36.432908 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:36.432815 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b948fdc0-b375-455d-8669-d47c9d84e8de-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7bc9z\" (UID: \"b948fdc0-b375-455d-8669-d47c9d84e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" Nov 04 04:54:36.432908 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:36.432843 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b948fdc0-b375-455d-8669-d47c9d84e8de-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7bc9z\" (UID: \"b948fdc0-b375-455d-8669-d47c9d84e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" Nov 04 04:54:36.432908 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:36.432866 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b948fdc0-b375-455d-8669-d47c9d84e8de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7bc9z\" (UID: \"b948fdc0-b375-455d-8669-d47c9d84e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" Nov 04 04:54:36.432908 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:36.432894 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b948fdc0-b375-455d-8669-d47c9d84e8de-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7bc9z\" (UID: \"b948fdc0-b375-455d-8669-d47c9d84e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" Nov 04 04:54:36.433206 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:36.432917 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b948fdc0-b375-455d-8669-d47c9d84e8de-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7bc9z\" (UID: \"b948fdc0-b375-455d-8669-d47c9d84e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" Nov 04 04:54:36.433206 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:36.433111 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b948fdc0-b375-455d-8669-d47c9d84e8de-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7bc9z\" (UID: \"b948fdc0-b375-455d-8669-d47c9d84e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" Nov 04 04:54:36.433206 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:36.433166 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b948fdc0-b375-455d-8669-d47c9d84e8de-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7bc9z\" (UID: \"b948fdc0-b375-455d-8669-d47c9d84e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" Nov 04 04:54:36.433397 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:36.433369 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b948fdc0-b375-455d-8669-d47c9d84e8de-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7bc9z\" (UID: \"b948fdc0-b375-455d-8669-d47c9d84e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" Nov 04 04:54:36.433397 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:36.433383 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b948fdc0-b375-455d-8669-d47c9d84e8de-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7bc9z\" (UID: \"b948fdc0-b375-455d-8669-d47c9d84e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" Nov 04 04:54:36.433705 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:36.433680 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b948fdc0-b375-455d-8669-d47c9d84e8de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7bc9z\" (UID: \"b948fdc0-b375-455d-8669-d47c9d84e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" Nov 04 04:54:36.435081 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:36.435065 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b948fdc0-b375-455d-8669-d47c9d84e8de-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7bc9z\" (UID: \"b948fdc0-b375-455d-8669-d47c9d84e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" Nov 04 04:54:36.435184 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:36.435166 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b948fdc0-b375-455d-8669-d47c9d84e8de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7bc9z\" (UID: \"b948fdc0-b375-455d-8669-d47c9d84e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" Nov 04 04:54:36.490189 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:36.490160 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b948fdc0-b375-455d-8669-d47c9d84e8de-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7bc9z\" (UID: \"b948fdc0-b375-455d-8669-d47c9d84e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" Nov 04 04:54:36.490673 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:36.490652 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpwdr\" (UniqueName: \"kubernetes.io/projected/b948fdc0-b375-455d-8669-d47c9d84e8de-kube-api-access-jpwdr\") pod \"maas-default-gateway-openshift-default-57f78bcd55-7bc9z\" (UID: \"b948fdc0-b375-455d-8669-d47c9d84e8de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" Nov 04 04:54:36.492269 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:36.492251 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" Nov 04 04:54:36.918184 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:36.918157 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z"] Nov 04 04:54:36.920033 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:54:36.919995 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb948fdc0_b375_455d_8669_d47c9d84e8de.slice/crio-21e16cbbf733603971babe0ec6960d935e9be6c82acd25146f821630165af781 WatchSource:0}: Error finding container 21e16cbbf733603971babe0ec6960d935e9be6c82acd25146f821630165af781: Status 404 returned error can't find the container with id 21e16cbbf733603971babe0ec6960d935e9be6c82acd25146f821630165af781 Nov 04 04:54:36.922196 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:36.922164 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 04:54:36.922329 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:36.922229 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 04:54:36.922329 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:36.922259 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 04:54:36.947354 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:36.947333 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" event={"ID":"b948fdc0-b375-455d-8669-d47c9d84e8de","Type":"ContainerStarted","Data":"21e16cbbf733603971babe0ec6960d935e9be6c82acd25146f821630165af781"} Nov 04 04:54:37.953732 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:37.953698 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" event={"ID":"b948fdc0-b375-455d-8669-d47c9d84e8de","Type":"ContainerStarted","Data":"f64e82fb7c2d93cf695b7f7bcbd70c13705673bfdc72c8d05e81ed87e482eed2"} Nov 04 04:54:38.166322 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:38.163184 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" podStartSLOduration=2.16316627 podStartE2EDuration="2.16316627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:54:38.112509452 +0000 UTC m=+519.802591899" watchObservedRunningTime="2026-11-04 04:54:38.16316627 +0000 UTC m=+519.853248717" Nov 04 04:54:38.493601 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:38.493563 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" Nov 04 04:54:38.494831 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:38.494799 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7bc9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 04 04:54:38.494961 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:38.494844 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" podUID="b948fdc0-b375-455d-8669-d47c9d84e8de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 04 04:54:39.493260 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:39.493227 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7bc9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 04 04:54:39.493673 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:39.493281 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" podUID="b948fdc0-b375-455d-8669-d47c9d84e8de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 04 04:54:40.493573 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:40.493540 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7bc9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 04 04:54:40.493971 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:40.493615 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" podUID="b948fdc0-b375-455d-8669-d47c9d84e8de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 04 04:54:41.493272 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:41.493234 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7bc9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 04 04:54:41.493464 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:41.493351 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" podUID="b948fdc0-b375-455d-8669-d47c9d84e8de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 04 04:54:42.493566 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:42.493527 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7bc9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 04 04:54:42.493971 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:42.493580 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" podUID="b948fdc0-b375-455d-8669-d47c9d84e8de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 04 04:54:42.940816 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:42.940783 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8bb5875fc-dct4m"] Nov 04 04:54:42.944377 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:42.944358 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8bb5875fc-dct4m" Nov 04 04:54:42.993153 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:42.993132 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6gwpp\"" Nov 04 04:54:42.993153 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:42.993141 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 04 04:54:43.003893 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:43.003873 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8bb5875fc-dct4m"] Nov 04 04:54:43.110615 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:43.110575 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/744859c8-b8e3-4fc5-b31c-de2bcec377dc-tls-cert\") pod \"authorino-8bb5875fc-dct4m\" (UID: \"744859c8-b8e3-4fc5-b31c-de2bcec377dc\") " pod="kuadrant-system/authorino-8bb5875fc-dct4m" Nov 04 04:54:43.110763 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:43.110687 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvt7f\" (UniqueName: \"kubernetes.io/projected/744859c8-b8e3-4fc5-b31c-de2bcec377dc-kube-api-access-rvt7f\") pod \"authorino-8bb5875fc-dct4m\" (UID: \"744859c8-b8e3-4fc5-b31c-de2bcec377dc\") " pod="kuadrant-system/authorino-8bb5875fc-dct4m" Nov 04 04:54:43.211176 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:43.211091 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/744859c8-b8e3-4fc5-b31c-de2bcec377dc-tls-cert\") pod \"authorino-8bb5875fc-dct4m\" (UID: \"744859c8-b8e3-4fc5-b31c-de2bcec377dc\") " pod="kuadrant-system/authorino-8bb5875fc-dct4m" Nov 04 04:54:43.211176 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:43.211147 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvt7f\" (UniqueName: \"kubernetes.io/projected/744859c8-b8e3-4fc5-b31c-de2bcec377dc-kube-api-access-rvt7f\") pod \"authorino-8bb5875fc-dct4m\" (UID: \"744859c8-b8e3-4fc5-b31c-de2bcec377dc\") " pod="kuadrant-system/authorino-8bb5875fc-dct4m" Nov 04 04:54:43.213636 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:43.213603 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/744859c8-b8e3-4fc5-b31c-de2bcec377dc-tls-cert\") pod \"authorino-8bb5875fc-dct4m\" (UID: \"744859c8-b8e3-4fc5-b31c-de2bcec377dc\") " pod="kuadrant-system/authorino-8bb5875fc-dct4m" Nov 04 04:54:43.268166 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:43.268140 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvt7f\" (UniqueName: \"kubernetes.io/projected/744859c8-b8e3-4fc5-b31c-de2bcec377dc-kube-api-access-rvt7f\") pod \"authorino-8bb5875fc-dct4m\" (UID: \"744859c8-b8e3-4fc5-b31c-de2bcec377dc\") " pod="kuadrant-system/authorino-8bb5875fc-dct4m" Nov 04 04:54:43.493109 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:43.493036 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7bc9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 04 04:54:43.493109 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:43.493082 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" podUID="b948fdc0-b375-455d-8669-d47c9d84e8de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 04 04:54:43.559241 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:43.559215 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8bb5875fc-dct4m" Nov 04 04:54:43.931276 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:43.931251 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8bb5875fc-dct4m"] Nov 04 04:54:43.934095 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:54:43.934065 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod744859c8_b8e3_4fc5_b31c_de2bcec377dc.slice/crio-bfe0ed20829e262afe0d667291caa9bf33c04ff4ffa83082a817cc729f385513 WatchSource:0}: Error finding container bfe0ed20829e262afe0d667291caa9bf33c04ff4ffa83082a817cc729f385513: Status 404 returned error can't find the container with id bfe0ed20829e262afe0d667291caa9bf33c04ff4ffa83082a817cc729f385513 Nov 04 04:54:44.006620 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:44.006593 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8bb5875fc-dct4m" event={"ID":"744859c8-b8e3-4fc5-b31c-de2bcec377dc","Type":"ContainerStarted","Data":"bfe0ed20829e262afe0d667291caa9bf33c04ff4ffa83082a817cc729f385513"} Nov 04 04:54:44.492681 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:44.492649 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-7bc9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 04 04:54:44.492841 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:44.492698 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" podUID="b948fdc0-b375-455d-8669-d47c9d84e8de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 04 04:54:45.497218 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:45.497191 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" Nov 04 04:54:45.497682 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:45.497496 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" Nov 04 04:54:45.498232 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:45.498214 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-7bc9z" Nov 04 04:54:46.016980 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:46.016941 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8bb5875fc-dct4m" event={"ID":"744859c8-b8e3-4fc5-b31c-de2bcec377dc","Type":"ContainerStarted","Data":"f996073abfba2b87df5abb4e4ae8bd1614caa4804580d9d0ea731478bae5ffe3"} Nov 04 04:54:46.153809 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:54:46.153746 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-8bb5875fc-dct4m" podStartSLOduration=2.552904089 podStartE2EDuration="4.15372767s" totalImagesPullingTime="1.600823581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:54:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:54:46.150931931 +0000 UTC m=+527.841014377" watchObservedRunningTime="2026-11-04 04:54:46.15372767 +0000 UTC m=+527.843810120" Nov 04 04:55:03.457678 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:55:03.457638 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 04:55:21.538397 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:21.538363 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz"] Nov 04 04:55:21.542152 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:21.542127 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" Nov 04 04:55:21.623953 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:21.623927 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbd01e49-964f-4c3f-a14b-249e4047fe34-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cd7gz\" (UID: \"fbd01e49-964f-4c3f-a14b-249e4047fe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" Nov 04 04:55:21.624059 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:21.623956 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbd01e49-964f-4c3f-a14b-249e4047fe34-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cd7gz\" (UID: \"fbd01e49-964f-4c3f-a14b-249e4047fe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" Nov 04 04:55:21.624059 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:21.623997 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbd01e49-964f-4c3f-a14b-249e4047fe34-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cd7gz\" (UID: \"fbd01e49-964f-4c3f-a14b-249e4047fe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" Nov 04 04:55:21.624145 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:21.624068 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbd01e49-964f-4c3f-a14b-249e4047fe34-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cd7gz\" (UID: \"fbd01e49-964f-4c3f-a14b-249e4047fe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" Nov 04 04:55:21.624145 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:21.624100 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd01e49-964f-4c3f-a14b-249e4047fe34-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cd7gz\" (UID: \"fbd01e49-964f-4c3f-a14b-249e4047fe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" Nov 04 04:55:21.624209 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:21.624144 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbd01e49-964f-4c3f-a14b-249e4047fe34-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cd7gz\" (UID: \"fbd01e49-964f-4c3f-a14b-249e4047fe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" Nov 04 04:55:21.624209 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:21.624168 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnpvc\" (UniqueName: \"kubernetes.io/projected/fbd01e49-964f-4c3f-a14b-249e4047fe34-kube-api-access-xnpvc\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cd7gz\" (UID: \"fbd01e49-964f-4c3f-a14b-249e4047fe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" Nov 04 04:55:21.624209 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:21.624188 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd01e49-964f-4c3f-a14b-249e4047fe34-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cd7gz\" (UID: \"fbd01e49-964f-4c3f-a14b-249e4047fe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" Nov 04 04:55:21.624323 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:21.624269 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbd01e49-964f-4c3f-a14b-249e4047fe34-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cd7gz\" (UID: \"fbd01e49-964f-4c3f-a14b-249e4047fe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" Nov 04 04:55:21.724986 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:21.724961 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbd01e49-964f-4c3f-a14b-249e4047fe34-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cd7gz\" (UID: \"fbd01e49-964f-4c3f-a14b-249e4047fe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" Nov 04 04:55:21.725181 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:21.725008 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbd01e49-964f-4c3f-a14b-249e4047fe34-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cd7gz\" (UID: \"fbd01e49-964f-4c3f-a14b-249e4047fe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" Nov 04 04:55:21.725181 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:21.725036 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbd01e49-964f-4c3f-a14b-249e4047fe34-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cd7gz\" (UID: \"fbd01e49-964f-4c3f-a14b-249e4047fe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" Nov 04 04:55:21.725181 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:21.725082 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbd01e49-964f-4c3f-a14b-249e4047fe34-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cd7gz\" (UID: \"fbd01e49-964f-4c3f-a14b-249e4047fe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" Nov 04 04:55:21.725181 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:21.725117 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbd01e49-964f-4c3f-a14b-249e4047fe34-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cd7gz\" (UID: \"fbd01e49-964f-4c3f-a14b-249e4047fe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" Nov 04 04:55:21.725181 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:21.725146 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd01e49-964f-4c3f-a14b-249e4047fe34-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cd7gz\" (UID: \"fbd01e49-964f-4c3f-a14b-249e4047fe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" Nov 04 04:55:21.725181 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:21.725169 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbd01e49-964f-4c3f-a14b-249e4047fe34-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cd7gz\" (UID: \"fbd01e49-964f-4c3f-a14b-249e4047fe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" Nov 04 04:55:21.725552 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:21.725206 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnpvc\" (UniqueName: \"kubernetes.io/projected/fbd01e49-964f-4c3f-a14b-249e4047fe34-kube-api-access-xnpvc\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cd7gz\" (UID: \"fbd01e49-964f-4c3f-a14b-249e4047fe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" Nov 04 04:55:21.725552 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:21.725235 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd01e49-964f-4c3f-a14b-249e4047fe34-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cd7gz\" (UID: \"fbd01e49-964f-4c3f-a14b-249e4047fe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" Nov 04 04:55:21.725552 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:21.725467 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbd01e49-964f-4c3f-a14b-249e4047fe34-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cd7gz\" (UID: \"fbd01e49-964f-4c3f-a14b-249e4047fe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" Nov 04 04:55:21.725698 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:21.725588 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd01e49-964f-4c3f-a14b-249e4047fe34-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cd7gz\" (UID: \"fbd01e49-964f-4c3f-a14b-249e4047fe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" Nov 04 04:55:21.725698 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:21.725605 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd01e49-964f-4c3f-a14b-249e4047fe34-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cd7gz\" (UID: \"fbd01e49-964f-4c3f-a14b-249e4047fe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" Nov 04 04:55:21.725792 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:21.725768 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbd01e49-964f-4c3f-a14b-249e4047fe34-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cd7gz\" (UID: \"fbd01e49-964f-4c3f-a14b-249e4047fe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" Nov 04 04:55:21.725963 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:21.725945 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbd01e49-964f-4c3f-a14b-249e4047fe34-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cd7gz\" (UID: \"fbd01e49-964f-4c3f-a14b-249e4047fe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" Nov 04 04:55:21.727362 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:21.727331 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbd01e49-964f-4c3f-a14b-249e4047fe34-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cd7gz\" (UID: \"fbd01e49-964f-4c3f-a14b-249e4047fe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" Nov 04 04:55:21.727558 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:21.727542 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbd01e49-964f-4c3f-a14b-249e4047fe34-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cd7gz\" (UID: \"fbd01e49-964f-4c3f-a14b-249e4047fe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" Nov 04 04:55:21.746340 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:21.746316 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz"] Nov 04 04:55:21.782008 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:21.781982 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbd01e49-964f-4c3f-a14b-249e4047fe34-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cd7gz\" (UID: \"fbd01e49-964f-4c3f-a14b-249e4047fe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" Nov 04 04:55:21.782099 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:21.782063 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnpvc\" (UniqueName: \"kubernetes.io/projected/fbd01e49-964f-4c3f-a14b-249e4047fe34-kube-api-access-xnpvc\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cd7gz\" (UID: \"fbd01e49-964f-4c3f-a14b-249e4047fe34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" Nov 04 04:55:21.854126 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:21.854104 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" Nov 04 04:55:22.285791 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:22.285746 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz"] Nov 04 04:55:22.288995 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:22.288946 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 04:55:22.289119 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:22.289069 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 04:55:22.289172 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:22.289150 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 04:55:23.206600 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:23.206556 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" event={"ID":"fbd01e49-964f-4c3f-a14b-249e4047fe34","Type":"ContainerStarted","Data":"89fae1e93f81d96e91a56b87a85ad5734cf30077bc0dd4c8c6af0c0b273d38a7"} Nov 04 04:55:23.206999 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:23.206616 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" event={"ID":"fbd01e49-964f-4c3f-a14b-249e4047fe34","Type":"ContainerStarted","Data":"95513cfd089cb7e9de0eeb4adacec12ba00ae6b61ca748b33cc86e909e263f48"} Nov 04 04:55:23.855236 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:23.855204 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" Nov 04 04:55:23.870317 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:23.856374 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-cd7gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 04 04:55:23.870317 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:23.856422 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" podUID="fbd01e49-964f-4c3f-a14b-249e4047fe34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 04 04:55:24.855307 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:24.855265 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-cd7gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 04 04:55:24.855676 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:24.855337 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" podUID="fbd01e49-964f-4c3f-a14b-249e4047fe34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 04 04:55:25.855406 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:25.855378 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-cd7gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 04 04:55:25.855766 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:25.855425 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" podUID="fbd01e49-964f-4c3f-a14b-249e4047fe34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 04 04:55:26.854874 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:26.854843 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-cd7gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 04 04:55:26.855042 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:26.854896 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" podUID="fbd01e49-964f-4c3f-a14b-249e4047fe34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 04 04:55:27.855003 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:27.854971 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-cd7gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 04 04:55:27.855478 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:27.855026 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" podUID="fbd01e49-964f-4c3f-a14b-249e4047fe34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 04 04:55:28.854883 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:28.854843 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-cd7gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 04 04:55:28.855074 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:28.854895 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" podUID="fbd01e49-964f-4c3f-a14b-249e4047fe34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 04 04:55:29.854516 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:29.854484 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-cd7gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 04 04:55:29.854676 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:29.854534 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" podUID="fbd01e49-964f-4c3f-a14b-249e4047fe34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 04 04:55:30.870369 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:30.870335 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" Nov 04 04:55:30.870839 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:30.870717 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" Nov 04 04:55:30.871489 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:30.871470 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" Nov 04 04:55:31.031609 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:31.031563 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" podStartSLOduration=10.031551174 podStartE2EDuration="10.031551174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:55:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:55:23.377148313 +0000 UTC m=+565.067230762" watchObservedRunningTime="2026-11-04 04:55:31.031551174 +0000 UTC m=+572.721633620" Nov 04 04:55:33.457596 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:55:33.457561 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 04:55:53.289424 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.289390 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj"] Nov 04 04:55:53.293186 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.293165 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" Nov 04 04:55:53.296727 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.296696 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5798fb27-7552-42f7-ae48-c4ff7decff83-workload-socket\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj\" (UID: \"5798fb27-7552-42f7-ae48-c4ff7decff83\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" Nov 04 04:55:53.296864 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.296758 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5798fb27-7552-42f7-ae48-c4ff7decff83-istio-data\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj\" (UID: \"5798fb27-7552-42f7-ae48-c4ff7decff83\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" Nov 04 04:55:53.296935 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.296856 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5798fb27-7552-42f7-ae48-c4ff7decff83-istio-podinfo\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj\" (UID: \"5798fb27-7552-42f7-ae48-c4ff7decff83\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" Nov 04 04:55:53.296935 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.296892 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5798fb27-7552-42f7-ae48-c4ff7decff83-istio-token\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj\" (UID: \"5798fb27-7552-42f7-ae48-c4ff7decff83\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" Nov 04 04:55:53.297009 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.296979 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5798fb27-7552-42f7-ae48-c4ff7decff83-credential-socket\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj\" (UID: \"5798fb27-7552-42f7-ae48-c4ff7decff83\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" Nov 04 04:55:53.297046 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.297007 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5798fb27-7552-42f7-ae48-c4ff7decff83-istiod-ca-cert\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj\" (UID: \"5798fb27-7552-42f7-ae48-c4ff7decff83\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" Nov 04 04:55:53.297046 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.297024 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5798fb27-7552-42f7-ae48-c4ff7decff83-istio-envoy\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj\" (UID: \"5798fb27-7552-42f7-ae48-c4ff7decff83\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" Nov 04 04:55:53.297162 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.297056 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5798fb27-7552-42f7-ae48-c4ff7decff83-workload-certs\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj\" (UID: \"5798fb27-7552-42f7-ae48-c4ff7decff83\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" Nov 04 04:55:53.297196 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.297176 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkjqb\" (UniqueName: \"kubernetes.io/projected/5798fb27-7552-42f7-ae48-c4ff7decff83-kube-api-access-tkjqb\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj\" (UID: \"5798fb27-7552-42f7-ae48-c4ff7decff83\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" Nov 04 04:55:53.342204 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.342172 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-a48f7e-openshift-default-dockercfg-tlcwj\"" Nov 04 04:55:53.396980 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.396949 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj"] Nov 04 04:55:53.398243 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.398199 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5798fb27-7552-42f7-ae48-c4ff7decff83-istio-data\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj\" (UID: \"5798fb27-7552-42f7-ae48-c4ff7decff83\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" Nov 04 04:55:53.398328 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.398272 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5798fb27-7552-42f7-ae48-c4ff7decff83-istio-podinfo\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj\" (UID: \"5798fb27-7552-42f7-ae48-c4ff7decff83\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" Nov 04 04:55:53.398328 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.398319 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5798fb27-7552-42f7-ae48-c4ff7decff83-istio-token\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj\" (UID: \"5798fb27-7552-42f7-ae48-c4ff7decff83\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" Nov 04 04:55:53.398444 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.398372 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5798fb27-7552-42f7-ae48-c4ff7decff83-credential-socket\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj\" (UID: \"5798fb27-7552-42f7-ae48-c4ff7decff83\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" Nov 04 04:55:53.398444 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.398415 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5798fb27-7552-42f7-ae48-c4ff7decff83-istiod-ca-cert\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj\" (UID: \"5798fb27-7552-42f7-ae48-c4ff7decff83\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" Nov 04 04:55:53.398535 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.398498 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5798fb27-7552-42f7-ae48-c4ff7decff83-istio-envoy\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj\" (UID: \"5798fb27-7552-42f7-ae48-c4ff7decff83\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" Nov 04 04:55:53.398594 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.398574 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5798fb27-7552-42f7-ae48-c4ff7decff83-workload-certs\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj\" (UID: \"5798fb27-7552-42f7-ae48-c4ff7decff83\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" Nov 04 04:55:53.398644 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.398613 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkjqb\" (UniqueName: \"kubernetes.io/projected/5798fb27-7552-42f7-ae48-c4ff7decff83-kube-api-access-tkjqb\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj\" (UID: \"5798fb27-7552-42f7-ae48-c4ff7decff83\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" Nov 04 04:55:53.398688 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.398648 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5798fb27-7552-42f7-ae48-c4ff7decff83-workload-socket\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj\" (UID: \"5798fb27-7552-42f7-ae48-c4ff7decff83\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" Nov 04 04:55:53.398949 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.398926 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5798fb27-7552-42f7-ae48-c4ff7decff83-istio-data\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj\" (UID: \"5798fb27-7552-42f7-ae48-c4ff7decff83\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" Nov 04 04:55:53.399213 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.399188 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5798fb27-7552-42f7-ae48-c4ff7decff83-istiod-ca-cert\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj\" (UID: \"5798fb27-7552-42f7-ae48-c4ff7decff83\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" Nov 04 04:55:53.399553 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.399528 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5798fb27-7552-42f7-ae48-c4ff7decff83-workload-certs\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj\" (UID: \"5798fb27-7552-42f7-ae48-c4ff7decff83\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" Nov 04 04:55:53.400066 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.400042 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5798fb27-7552-42f7-ae48-c4ff7decff83-workload-socket\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj\" (UID: \"5798fb27-7552-42f7-ae48-c4ff7decff83\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" Nov 04 04:55:53.400211 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.400159 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5798fb27-7552-42f7-ae48-c4ff7decff83-credential-socket\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj\" (UID: \"5798fb27-7552-42f7-ae48-c4ff7decff83\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" Nov 04 04:55:53.401559 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.401535 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5798fb27-7552-42f7-ae48-c4ff7decff83-istio-podinfo\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj\" (UID: \"5798fb27-7552-42f7-ae48-c4ff7decff83\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" Nov 04 04:55:53.401731 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.401713 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5798fb27-7552-42f7-ae48-c4ff7decff83-istio-envoy\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj\" (UID: \"5798fb27-7552-42f7-ae48-c4ff7decff83\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" Nov 04 04:55:53.455292 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.455267 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5798fb27-7552-42f7-ae48-c4ff7decff83-istio-token\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj\" (UID: \"5798fb27-7552-42f7-ae48-c4ff7decff83\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" Nov 04 04:55:53.455616 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.455597 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkjqb\" (UniqueName: \"kubernetes.io/projected/5798fb27-7552-42f7-ae48-c4ff7decff83-kube-api-access-tkjqb\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj\" (UID: \"5798fb27-7552-42f7-ae48-c4ff7decff83\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" Nov 04 04:55:53.606429 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.606403 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" Nov 04 04:55:53.879591 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.879519 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7"] Nov 04 04:55:53.884139 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.884117 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" Nov 04 04:55:53.904004 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.903978 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-credential-socket\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7\" (UID: \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" Nov 04 04:55:53.904122 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.904015 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-istio-envoy\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7\" (UID: \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" Nov 04 04:55:53.904181 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.904162 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-istio-data\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7\" (UID: \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" Nov 04 04:55:53.904244 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.904227 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-istio-podinfo\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7\" (UID: \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" Nov 04 04:55:53.904285 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.904260 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scrtb\" (UniqueName: \"kubernetes.io/projected/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-kube-api-access-scrtb\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7\" (UID: \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" Nov 04 04:55:53.904362 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.904351 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-workload-socket\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7\" (UID: \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" Nov 04 04:55:53.904402 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.904371 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-istiod-ca-cert\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7\" (UID: \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" Nov 04 04:55:53.904432 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.904401 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-workload-certs\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7\" (UID: \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" Nov 04 04:55:53.904519 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.904501 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-istio-token\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7\" (UID: \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" Nov 04 04:55:53.932725 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.932704 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-204e38-openshift-default-dockercfg-ftp67\"" Nov 04 04:55:53.943542 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:53.943520 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7"] Nov 04 04:55:54.005789 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.005749 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-workload-socket\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7\" (UID: \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" Nov 04 04:55:54.005931 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.005797 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-istiod-ca-cert\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7\" (UID: \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" Nov 04 04:55:54.005931 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.005840 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-workload-certs\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7\" (UID: \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" Nov 04 04:55:54.005931 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.005899 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-istio-token\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7\" (UID: \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" Nov 04 04:55:54.006096 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.006018 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-credential-socket\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7\" (UID: \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" Nov 04 04:55:54.006096 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.006064 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-istio-envoy\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7\" (UID: \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" Nov 04 04:55:54.006209 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.006096 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-workload-socket\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7\" (UID: \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" Nov 04 04:55:54.006209 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.006152 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-istio-data\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7\" (UID: \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" Nov 04 04:55:54.006209 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.006191 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-workload-certs\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7\" (UID: \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" Nov 04 04:55:54.006209 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.006204 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-istio-podinfo\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7\" (UID: \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" Nov 04 04:55:54.006395 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.006252 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scrtb\" (UniqueName: \"kubernetes.io/projected/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-kube-api-access-scrtb\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7\" (UID: \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" Nov 04 04:55:54.006443 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.006411 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-credential-socket\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7\" (UID: \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" Nov 04 04:55:54.006486 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.006473 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-istio-data\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7\" (UID: \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" Nov 04 04:55:54.006653 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.006637 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-istiod-ca-cert\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7\" (UID: \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" Nov 04 04:55:54.008273 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.008256 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-istio-envoy\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7\" (UID: \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" Nov 04 04:55:54.008403 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.008387 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-istio-podinfo\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7\" (UID: \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" Nov 04 04:55:54.047066 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.047037 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj"] Nov 04 04:55:54.049836 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:55:54.049806 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5798fb27_7552_42f7_ae48_c4ff7decff83.slice/crio-7fec7cd4e3e7b21e6ce146a595f49a7786b65ef8aa19b0bde9a93c1caf36fdc4 WatchSource:0}: Error finding container 7fec7cd4e3e7b21e6ce146a595f49a7786b65ef8aa19b0bde9a93c1caf36fdc4: Status 404 returned error can't find the container with id 7fec7cd4e3e7b21e6ce146a595f49a7786b65ef8aa19b0bde9a93c1caf36fdc4 Nov 04 04:55:54.052627 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.052595 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 04:55:54.052723 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.052671 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 04:55:54.052723 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.052718 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 04:55:54.063969 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.063927 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-istio-token\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7\" (UID: \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" Nov 04 04:55:54.065062 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.065039 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scrtb\" (UniqueName: \"kubernetes.io/projected/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-kube-api-access-scrtb\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7\" (UID: \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" Nov 04 04:55:54.199500 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.199416 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" Nov 04 04:55:54.359150 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.359114 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" event={"ID":"5798fb27-7552-42f7-ae48-c4ff7decff83","Type":"ContainerStarted","Data":"f9e1e716d3e3904299a6cf769a4559d76a778f432282c57bea1b14556a26994b"} Nov 04 04:55:54.359678 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.359176 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" event={"ID":"5798fb27-7552-42f7-ae48-c4ff7decff83","Type":"ContainerStarted","Data":"7fec7cd4e3e7b21e6ce146a595f49a7786b65ef8aa19b0bde9a93c1caf36fdc4"} Nov 04 04:55:54.475076 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.474985 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" podStartSLOduration=1.474972729 podStartE2EDuration="1.474972729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:55:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:55:54.472377211 +0000 UTC m=+596.162459660" watchObservedRunningTime="2026-11-04 04:55:54.474972729 +0000 UTC m=+596.165055176" Nov 04 04:55:54.592954 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.592725 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7"] Nov 04 04:55:54.598183 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:55:54.598154 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e150c0f_44ce_4946_ac17_bbf8ac9bfbb3.slice/crio-f02ee307fd68d360dad77af5981bdda36587a2999dea67354fbec78e7a487869 WatchSource:0}: Error finding container f02ee307fd68d360dad77af5981bdda36587a2999dea67354fbec78e7a487869: Status 404 returned error can't find the container with id f02ee307fd68d360dad77af5981bdda36587a2999dea67354fbec78e7a487869 Nov 04 04:55:54.600223 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.600189 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 04:55:54.600349 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.600254 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 04:55:54.600349 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.600284 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 04:55:54.607356 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.607337 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" Nov 04 04:55:54.825330 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.825288 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468"] Nov 04 04:55:54.828918 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.828898 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" Nov 04 04:55:54.877605 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.877569 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-db8fae-openshift-default-dockercfg-l9grm\"" Nov 04 04:55:54.891398 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.891369 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468"] Nov 04 04:55:54.916541 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.916511 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24ea2807-ab22-495c-b66a-b35111e36b97-istiod-ca-cert\") pod \"e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468\" (UID: \"24ea2807-ab22-495c-b66a-b35111e36b97\") " pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" Nov 04 04:55:54.916686 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.916551 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24ea2807-ab22-495c-b66a-b35111e36b97-istio-token\") pod \"e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468\" (UID: \"24ea2807-ab22-495c-b66a-b35111e36b97\") " pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" Nov 04 04:55:54.916686 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.916581 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24ea2807-ab22-495c-b66a-b35111e36b97-workload-certs\") pod \"e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468\" (UID: \"24ea2807-ab22-495c-b66a-b35111e36b97\") " pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" Nov 04 04:55:54.916772 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.916736 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24ea2807-ab22-495c-b66a-b35111e36b97-credential-socket\") pod \"e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468\" (UID: \"24ea2807-ab22-495c-b66a-b35111e36b97\") " pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" Nov 04 04:55:54.916816 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.916801 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24ea2807-ab22-495c-b66a-b35111e36b97-workload-socket\") pod \"e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468\" (UID: \"24ea2807-ab22-495c-b66a-b35111e36b97\") " pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" Nov 04 04:55:54.916917 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.916882 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7ph4\" (UniqueName: \"kubernetes.io/projected/24ea2807-ab22-495c-b66a-b35111e36b97-kube-api-access-w7ph4\") pod \"e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468\" (UID: \"24ea2807-ab22-495c-b66a-b35111e36b97\") " pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" Nov 04 04:55:54.916961 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.916942 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24ea2807-ab22-495c-b66a-b35111e36b97-istio-data\") pod \"e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468\" (UID: \"24ea2807-ab22-495c-b66a-b35111e36b97\") " pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" Nov 04 04:55:54.916995 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.916968 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24ea2807-ab22-495c-b66a-b35111e36b97-istio-podinfo\") pod \"e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468\" (UID: \"24ea2807-ab22-495c-b66a-b35111e36b97\") " pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" Nov 04 04:55:54.917038 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:54.917016 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24ea2807-ab22-495c-b66a-b35111e36b97-istio-envoy\") pod \"e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468\" (UID: \"24ea2807-ab22-495c-b66a-b35111e36b97\") " pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" Nov 04 04:55:55.018284 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.018258 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7ph4\" (UniqueName: \"kubernetes.io/projected/24ea2807-ab22-495c-b66a-b35111e36b97-kube-api-access-w7ph4\") pod \"e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468\" (UID: \"24ea2807-ab22-495c-b66a-b35111e36b97\") " pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" Nov 04 04:55:55.018451 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.018292 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24ea2807-ab22-495c-b66a-b35111e36b97-istio-data\") pod \"e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468\" (UID: \"24ea2807-ab22-495c-b66a-b35111e36b97\") " pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" Nov 04 04:55:55.018451 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.018340 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24ea2807-ab22-495c-b66a-b35111e36b97-istio-podinfo\") pod \"e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468\" (UID: \"24ea2807-ab22-495c-b66a-b35111e36b97\") " pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" Nov 04 04:55:55.018451 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.018370 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24ea2807-ab22-495c-b66a-b35111e36b97-istio-envoy\") pod \"e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468\" (UID: \"24ea2807-ab22-495c-b66a-b35111e36b97\") " pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" Nov 04 04:55:55.018451 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.018428 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24ea2807-ab22-495c-b66a-b35111e36b97-istiod-ca-cert\") pod \"e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468\" (UID: \"24ea2807-ab22-495c-b66a-b35111e36b97\") " pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" Nov 04 04:55:55.018451 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.018450 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24ea2807-ab22-495c-b66a-b35111e36b97-istio-token\") pod \"e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468\" (UID: \"24ea2807-ab22-495c-b66a-b35111e36b97\") " pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" Nov 04 04:55:55.018715 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.018561 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24ea2807-ab22-495c-b66a-b35111e36b97-workload-certs\") pod \"e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468\" (UID: \"24ea2807-ab22-495c-b66a-b35111e36b97\") " pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" Nov 04 04:55:55.018715 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.018606 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24ea2807-ab22-495c-b66a-b35111e36b97-credential-socket\") pod \"e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468\" (UID: \"24ea2807-ab22-495c-b66a-b35111e36b97\") " pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" Nov 04 04:55:55.018715 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.018659 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24ea2807-ab22-495c-b66a-b35111e36b97-workload-socket\") pod \"e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468\" (UID: \"24ea2807-ab22-495c-b66a-b35111e36b97\") " pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" Nov 04 04:55:55.018869 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.018796 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24ea2807-ab22-495c-b66a-b35111e36b97-istio-data\") pod \"e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468\" (UID: \"24ea2807-ab22-495c-b66a-b35111e36b97\") " pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" Nov 04 04:55:55.018927 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.018888 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24ea2807-ab22-495c-b66a-b35111e36b97-workload-certs\") pod \"e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468\" (UID: \"24ea2807-ab22-495c-b66a-b35111e36b97\") " pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" Nov 04 04:55:55.019002 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.018979 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24ea2807-ab22-495c-b66a-b35111e36b97-credential-socket\") pod \"e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468\" (UID: \"24ea2807-ab22-495c-b66a-b35111e36b97\") " pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" Nov 04 04:55:55.019090 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.019071 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24ea2807-ab22-495c-b66a-b35111e36b97-workload-socket\") pod \"e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468\" (UID: \"24ea2807-ab22-495c-b66a-b35111e36b97\") " pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" Nov 04 04:55:55.019427 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.019411 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24ea2807-ab22-495c-b66a-b35111e36b97-istiod-ca-cert\") pod \"e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468\" (UID: \"24ea2807-ab22-495c-b66a-b35111e36b97\") " pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" Nov 04 04:55:55.020769 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.020745 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24ea2807-ab22-495c-b66a-b35111e36b97-istio-envoy\") pod \"e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468\" (UID: \"24ea2807-ab22-495c-b66a-b35111e36b97\") " pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" Nov 04 04:55:55.020965 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.020949 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24ea2807-ab22-495c-b66a-b35111e36b97-istio-podinfo\") pod \"e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468\" (UID: \"24ea2807-ab22-495c-b66a-b35111e36b97\") " pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" Nov 04 04:55:55.075891 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.075827 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24ea2807-ab22-495c-b66a-b35111e36b97-istio-token\") pod \"e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468\" (UID: \"24ea2807-ab22-495c-b66a-b35111e36b97\") " pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" Nov 04 04:55:55.076096 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.076077 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7ph4\" (UniqueName: \"kubernetes.io/projected/24ea2807-ab22-495c-b66a-b35111e36b97-kube-api-access-w7ph4\") pod \"e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468\" (UID: \"24ea2807-ab22-495c-b66a-b35111e36b97\") " pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" Nov 04 04:55:55.115103 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.115078 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl"] Nov 04 04:55:55.120009 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.119989 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" Nov 04 04:55:55.142788 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.142764 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" Nov 04 04:55:55.179185 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.179151 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl"] Nov 04 04:55:55.220023 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.219997 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/518e3ddc-9b6e-4127-a47c-397d9117a3ae-istiod-ca-cert\") pod \"e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl\" (UID: \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\") " pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" Nov 04 04:55:55.220147 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.220044 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/518e3ddc-9b6e-4127-a47c-397d9117a3ae-istio-token\") pod \"e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl\" (UID: \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\") " pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" Nov 04 04:55:55.220147 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.220129 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/518e3ddc-9b6e-4127-a47c-397d9117a3ae-istio-envoy\") pod \"e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl\" (UID: \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\") " pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" Nov 04 04:55:55.220262 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.220233 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/518e3ddc-9b6e-4127-a47c-397d9117a3ae-istio-podinfo\") pod \"e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl\" (UID: \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\") " pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" Nov 04 04:55:55.220329 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.220286 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/518e3ddc-9b6e-4127-a47c-397d9117a3ae-credential-socket\") pod \"e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl\" (UID: \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\") " pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" Nov 04 04:55:55.220388 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.220373 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/518e3ddc-9b6e-4127-a47c-397d9117a3ae-workload-certs\") pod \"e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl\" (UID: \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\") " pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" Nov 04 04:55:55.220427 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.220401 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/518e3ddc-9b6e-4127-a47c-397d9117a3ae-istio-data\") pod \"e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl\" (UID: \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\") " pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" Nov 04 04:55:55.220473 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.220425 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn7z9\" (UniqueName: \"kubernetes.io/projected/518e3ddc-9b6e-4127-a47c-397d9117a3ae-kube-api-access-bn7z9\") pod \"e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl\" (UID: \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\") " pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" Nov 04 04:55:55.220473 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.220444 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/518e3ddc-9b6e-4127-a47c-397d9117a3ae-workload-socket\") pod \"e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl\" (UID: \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\") " pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" Nov 04 04:55:55.256027 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.256006 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-e62689-openshift-default-dockercfg-qvbw7\"" Nov 04 04:55:55.321370 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.321332 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/518e3ddc-9b6e-4127-a47c-397d9117a3ae-istiod-ca-cert\") pod \"e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl\" (UID: \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\") " pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" Nov 04 04:55:55.321564 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.321406 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/518e3ddc-9b6e-4127-a47c-397d9117a3ae-istio-token\") pod \"e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl\" (UID: \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\") " pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" Nov 04 04:55:55.321564 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.321451 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/518e3ddc-9b6e-4127-a47c-397d9117a3ae-istio-envoy\") pod \"e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl\" (UID: \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\") " pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" Nov 04 04:55:55.321564 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.321483 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/518e3ddc-9b6e-4127-a47c-397d9117a3ae-istio-podinfo\") pod \"e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl\" (UID: \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\") " pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" Nov 04 04:55:55.321564 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.321517 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/518e3ddc-9b6e-4127-a47c-397d9117a3ae-credential-socket\") pod \"e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl\" (UID: \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\") " pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" Nov 04 04:55:55.321791 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.321582 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/518e3ddc-9b6e-4127-a47c-397d9117a3ae-workload-certs\") pod \"e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl\" (UID: \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\") " pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" Nov 04 04:55:55.321791 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.321618 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/518e3ddc-9b6e-4127-a47c-397d9117a3ae-istio-data\") pod \"e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl\" (UID: \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\") " pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" Nov 04 04:55:55.321791 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.321652 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn7z9\" (UniqueName: \"kubernetes.io/projected/518e3ddc-9b6e-4127-a47c-397d9117a3ae-kube-api-access-bn7z9\") pod \"e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl\" (UID: \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\") " pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" Nov 04 04:55:55.321791 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.321683 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/518e3ddc-9b6e-4127-a47c-397d9117a3ae-workload-socket\") pod \"e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl\" (UID: \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\") " pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" Nov 04 04:55:55.322042 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.322013 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/518e3ddc-9b6e-4127-a47c-397d9117a3ae-workload-certs\") pod \"e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl\" (UID: \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\") " pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" Nov 04 04:55:55.322196 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.322069 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/518e3ddc-9b6e-4127-a47c-397d9117a3ae-workload-socket\") pod \"e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl\" (UID: \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\") " pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" Nov 04 04:55:55.322196 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.322122 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/518e3ddc-9b6e-4127-a47c-397d9117a3ae-istio-data\") pod \"e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl\" (UID: \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\") " pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" Nov 04 04:55:55.322358 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.322338 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/518e3ddc-9b6e-4127-a47c-397d9117a3ae-credential-socket\") pod \"e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl\" (UID: \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\") " pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" Nov 04 04:55:55.322428 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.322373 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/518e3ddc-9b6e-4127-a47c-397d9117a3ae-istiod-ca-cert\") pod \"e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl\" (UID: \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\") " pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" Nov 04 04:55:55.324101 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.324074 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/518e3ddc-9b6e-4127-a47c-397d9117a3ae-istio-envoy\") pod \"e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl\" (UID: \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\") " pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" Nov 04 04:55:55.324399 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.324377 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/518e3ddc-9b6e-4127-a47c-397d9117a3ae-istio-podinfo\") pod \"e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl\" (UID: \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\") " pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" Nov 04 04:55:55.366200 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.366098 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" event={"ID":"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3","Type":"ContainerStarted","Data":"96dc40bd86256d67a635da6966dd828d2c6f96b60a30da6fd1ff64b4a5e007a0"} Nov 04 04:55:55.366638 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.366390 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" event={"ID":"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3","Type":"ContainerStarted","Data":"f02ee307fd68d360dad77af5981bdda36587a2999dea67354fbec78e7a487869"} Nov 04 04:55:55.379048 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.379025 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/518e3ddc-9b6e-4127-a47c-397d9117a3ae-istio-token\") pod \"e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl\" (UID: \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\") " pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" Nov 04 04:55:55.379216 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.379195 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn7z9\" (UniqueName: \"kubernetes.io/projected/518e3ddc-9b6e-4127-a47c-397d9117a3ae-kube-api-access-bn7z9\") pod \"e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl\" (UID: \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\") " pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" Nov 04 04:55:55.433170 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.433150 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" Nov 04 04:55:55.480531 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.480484 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" podStartSLOduration=2.480469293 podStartE2EDuration="2.480469293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:55:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:55:55.478162803 +0000 UTC m=+597.168245255" watchObservedRunningTime="2026-11-04 04:55:55.480469293 +0000 UTC m=+597.170551739" Nov 04 04:55:55.527411 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.527384 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468"] Nov 04 04:55:55.529579 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:55:55.529545 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ea2807_ab22_495c_b66a_b35111e36b97.slice/crio-06f7d5e6df70e090521670fa4b2aded7c92398d7277da075b5e41408ca570225 WatchSource:0}: Error finding container 06f7d5e6df70e090521670fa4b2aded7c92398d7277da075b5e41408ca570225: Status 404 returned error can't find the container with id 06f7d5e6df70e090521670fa4b2aded7c92398d7277da075b5e41408ca570225 Nov 04 04:55:55.532466 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.532431 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 04:55:55.532564 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.532499 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 04:55:55.532564 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.532530 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 04:55:55.612719 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.612691 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" Nov 04 04:55:55.815703 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.815682 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl"] Nov 04 04:55:55.817891 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:55:55.817864 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod518e3ddc_9b6e_4127_a47c_397d9117a3ae.slice/crio-b186a0d838fc5addb967d5882ad892985e9f21880c400272863334404e133964 WatchSource:0}: Error finding container b186a0d838fc5addb967d5882ad892985e9f21880c400272863334404e133964: Status 404 returned error can't find the container with id b186a0d838fc5addb967d5882ad892985e9f21880c400272863334404e133964 Nov 04 04:55:55.820252 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.820221 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 04:55:55.820333 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.820282 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 04:55:55.820333 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:55.820322 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 04:55:56.199995 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:56.199910 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" Nov 04 04:55:56.204975 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:56.204952 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" Nov 04 04:55:56.373489 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:56.373443 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" event={"ID":"24ea2807-ab22-495c-b66a-b35111e36b97","Type":"ContainerStarted","Data":"fb40a3de103e8fda221f07cce71ea4956f46b331866584e14b44b2409a8dc3d0"} Nov 04 04:55:56.373944 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:56.373516 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" event={"ID":"24ea2807-ab22-495c-b66a-b35111e36b97","Type":"ContainerStarted","Data":"06f7d5e6df70e090521670fa4b2aded7c92398d7277da075b5e41408ca570225"} Nov 04 04:55:56.375115 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:56.375084 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" event={"ID":"518e3ddc-9b6e-4127-a47c-397d9117a3ae","Type":"ContainerStarted","Data":"43742d519f2ebe3a7ef9a14ded98694cd286e629d0ed8159c3b8a4adf0cc81d4"} Nov 04 04:55:56.375267 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:56.375127 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" event={"ID":"518e3ddc-9b6e-4127-a47c-397d9117a3ae","Type":"ContainerStarted","Data":"b186a0d838fc5addb967d5882ad892985e9f21880c400272863334404e133964"} Nov 04 04:55:56.375391 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:56.375353 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" Nov 04 04:55:56.375636 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:56.375623 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" Nov 04 04:55:56.376185 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:56.376166 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" Nov 04 04:55:56.376378 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:56.376362 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" Nov 04 04:55:56.433368 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:56.433329 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" Nov 04 04:55:56.438181 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:56.438160 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" Nov 04 04:55:56.487223 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:56.487148 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" podStartSLOduration=2.4871371030000002 podStartE2EDuration="2.487137103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:55:56.486245994 +0000 UTC m=+598.176328441" watchObservedRunningTime="2026-11-04 04:55:56.487137103 +0000 UTC m=+598.177219550" Nov 04 04:55:56.827822 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:56.827778 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" podStartSLOduration=1.8277638619999999 podStartE2EDuration="1.827763862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:55:56.825788467 +0000 UTC m=+598.515870927" watchObservedRunningTime="2026-11-04 04:55:56.827763862 +0000 UTC m=+598.517846376" Nov 04 04:55:57.143433 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:57.143360 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" Nov 04 04:55:57.148515 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:57.148493 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" Nov 04 04:55:57.381036 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:57.380993 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" Nov 04 04:55:57.381518 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:57.381086 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" Nov 04 04:55:57.381623 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:57.381607 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" Nov 04 04:55:57.382028 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:55:57.382007 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" Nov 04 04:56:03.456734 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:56:03.456693 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 04:56:10.616232 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:10.616134 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-204e38-7479dcc588-qhznn"] Nov 04 04:56:10.654855 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:10.654825 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-204e38-7479dcc588-qhznn" Nov 04 04:56:10.678429 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:10.678405 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-204e38-7479dcc588-qhznn"] Nov 04 04:56:10.722270 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:10.722249 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-204e38\"" Nov 04 04:56:10.722427 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:10.722361 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-204e38-dockercfg-zswrj\"" Nov 04 04:56:10.773589 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:10.773563 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b78fb66-665e-454a-bc99-7331cfaf9bdc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-204e38-7479dcc588-qhznn\" (UID: \"1b78fb66-665e-454a-bc99-7331cfaf9bdc\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-204e38-7479dcc588-qhznn" Nov 04 04:56:10.773700 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:10.773638 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc46b\" (UniqueName: \"kubernetes.io/projected/1b78fb66-665e-454a-bc99-7331cfaf9bdc-kube-api-access-sc46b\") pod \"payload-processing-e2e-worker-w3-204e38-7479dcc588-qhznn\" (UID: \"1b78fb66-665e-454a-bc99-7331cfaf9bdc\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-204e38-7479dcc588-qhznn" Nov 04 04:56:10.874643 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:10.874573 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b78fb66-665e-454a-bc99-7331cfaf9bdc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-204e38-7479dcc588-qhznn\" (UID: \"1b78fb66-665e-454a-bc99-7331cfaf9bdc\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-204e38-7479dcc588-qhznn" Nov 04 04:56:10.874771 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:10.874662 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc46b\" (UniqueName: \"kubernetes.io/projected/1b78fb66-665e-454a-bc99-7331cfaf9bdc-kube-api-access-sc46b\") pod \"payload-processing-e2e-worker-w3-204e38-7479dcc588-qhznn\" (UID: \"1b78fb66-665e-454a-bc99-7331cfaf9bdc\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-204e38-7479dcc588-qhznn" Nov 04 04:56:10.875203 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:10.875185 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b78fb66-665e-454a-bc99-7331cfaf9bdc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-204e38-7479dcc588-qhznn\" (UID: \"1b78fb66-665e-454a-bc99-7331cfaf9bdc\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-204e38-7479dcc588-qhznn" Nov 04 04:56:10.931912 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:10.931889 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc46b\" (UniqueName: \"kubernetes.io/projected/1b78fb66-665e-454a-bc99-7331cfaf9bdc-kube-api-access-sc46b\") pod \"payload-processing-e2e-worker-w3-204e38-7479dcc588-qhznn\" (UID: \"1b78fb66-665e-454a-bc99-7331cfaf9bdc\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-204e38-7479dcc588-qhznn" Nov 04 04:56:10.970649 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:10.970629 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-204e38-7479dcc588-qhznn" Nov 04 04:56:11.343326 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:11.343292 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-204e38-7479dcc588-qhznn"] Nov 04 04:56:11.344910 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:56:11.344881 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b78fb66_665e_454a_bc99_7331cfaf9bdc.slice/crio-626f46310121904c4be0d621972c3f3ba49f169dfd1d5321e343800681577c6d WatchSource:0}: Error finding container 626f46310121904c4be0d621972c3f3ba49f169dfd1d5321e343800681577c6d: Status 404 returned error can't find the container with id 626f46310121904c4be0d621972c3f3ba49f169dfd1d5321e343800681577c6d Nov 04 04:56:11.458830 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:11.458801 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-204e38-7479dcc588-qhznn" event={"ID":"1b78fb66-665e-454a-bc99-7331cfaf9bdc","Type":"ContainerStarted","Data":"626f46310121904c4be0d621972c3f3ba49f169dfd1d5321e343800681577c6d"} Nov 04 04:56:14.480006 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:14.479919 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-204e38-7479dcc588-qhznn" event={"ID":"1b78fb66-665e-454a-bc99-7331cfaf9bdc","Type":"ContainerStarted","Data":"ec74d9dc4f9dc53e081e51ed53ea9fa1c81f164e14eef4711e10e643741bfd38"} Nov 04 04:56:14.480006 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:14.479985 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-204e38-7479dcc588-qhznn" Nov 04 04:56:14.589072 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:14.589022 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-204e38-7479dcc588-qhznn" podStartSLOduration=1.796335792 podStartE2EDuration="4.589008986s" totalImagesPullingTime="2.792673194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:56:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:56:14.587606705 +0000 UTC m=+616.277689152" watchObservedRunningTime="2026-11-04 04:56:14.589008986 +0000 UTC m=+616.279091435" Nov 04 04:56:18.794736 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:18.794696 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-19bf5b-7c6d784c4-s8rpn"] Nov 04 04:56:18.799165 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:18.799144 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-19bf5b-7c6d784c4-s8rpn" Nov 04 04:56:18.847807 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:18.847785 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-19bf5b\"" Nov 04 04:56:18.894097 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:18.894075 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-19bf5b-dockercfg-28b92\"" Nov 04 04:56:18.896266 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:18.896244 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-19bf5b-7c6d784c4-s8rpn"] Nov 04 04:56:18.969188 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:18.969153 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-854x4\" (UniqueName: \"kubernetes.io/projected/5368507f-308f-447a-8da1-f6fc28c7a88d-kube-api-access-854x4\") pod \"payload-processing-e2e-worker-w5-19bf5b-7c6d784c4-s8rpn\" (UID: \"5368507f-308f-447a-8da1-f6fc28c7a88d\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-19bf5b-7c6d784c4-s8rpn" Nov 04 04:56:18.969366 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:18.969224 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5368507f-308f-447a-8da1-f6fc28c7a88d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-19bf5b-7c6d784c4-s8rpn\" (UID: \"5368507f-308f-447a-8da1-f6fc28c7a88d\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-19bf5b-7c6d784c4-s8rpn" Nov 04 04:56:19.070288 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:19.070226 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-854x4\" (UniqueName: \"kubernetes.io/projected/5368507f-308f-447a-8da1-f6fc28c7a88d-kube-api-access-854x4\") pod \"payload-processing-e2e-worker-w5-19bf5b-7c6d784c4-s8rpn\" (UID: \"5368507f-308f-447a-8da1-f6fc28c7a88d\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-19bf5b-7c6d784c4-s8rpn" Nov 04 04:56:19.070288 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:19.070269 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5368507f-308f-447a-8da1-f6fc28c7a88d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-19bf5b-7c6d784c4-s8rpn\" (UID: \"5368507f-308f-447a-8da1-f6fc28c7a88d\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-19bf5b-7c6d784c4-s8rpn" Nov 04 04:56:19.070932 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:19.070913 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5368507f-308f-447a-8da1-f6fc28c7a88d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-19bf5b-7c6d784c4-s8rpn\" (UID: \"5368507f-308f-447a-8da1-f6fc28c7a88d\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-19bf5b-7c6d784c4-s8rpn" Nov 04 04:56:19.127817 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:19.127788 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-854x4\" (UniqueName: \"kubernetes.io/projected/5368507f-308f-447a-8da1-f6fc28c7a88d-kube-api-access-854x4\") pod \"payload-processing-e2e-worker-w5-19bf5b-7c6d784c4-s8rpn\" (UID: \"5368507f-308f-447a-8da1-f6fc28c7a88d\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-19bf5b-7c6d784c4-s8rpn" Nov 04 04:56:19.412970 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:19.412884 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-19bf5b-7c6d784c4-s8rpn" Nov 04 04:56:19.796333 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:19.796282 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-19bf5b-7c6d784c4-s8rpn"] Nov 04 04:56:19.798669 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:56:19.798642 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5368507f_308f_447a_8da1_f6fc28c7a88d.slice/crio-7a5057fd640abb02713712d52a27d552bce26a5c04428b1b63f2562bb0709dad WatchSource:0}: Error finding container 7a5057fd640abb02713712d52a27d552bce26a5c04428b1b63f2562bb0709dad: Status 404 returned error can't find the container with id 7a5057fd640abb02713712d52a27d552bce26a5c04428b1b63f2562bb0709dad Nov 04 04:56:20.544386 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:20.544342 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-19bf5b-7c6d784c4-s8rpn" event={"ID":"5368507f-308f-447a-8da1-f6fc28c7a88d","Type":"ContainerStarted","Data":"f4dea3bbdf37275210e082ce77cb010cec001b882af44c0ad0ee269f6661549f"} Nov 04 04:56:20.544585 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:20.544421 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-19bf5b-7c6d784c4-s8rpn" Nov 04 04:56:20.544585 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:20.544444 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-19bf5b-7c6d784c4-s8rpn" event={"ID":"5368507f-308f-447a-8da1-f6fc28c7a88d","Type":"ContainerStarted","Data":"7a5057fd640abb02713712d52a27d552bce26a5c04428b1b63f2562bb0709dad"} Nov 04 04:56:20.655704 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:20.655650 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-19bf5b-7c6d784c4-s8rpn" podStartSLOduration=2.655635494 podStartE2EDuration="2.655635494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:56:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:56:20.653492784 +0000 UTC m=+622.343575231" watchObservedRunningTime="2026-11-04 04:56:20.655635494 +0000 UTC m=+622.345718000" Nov 04 04:56:25.489015 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:25.488982 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-204e38-7479dcc588-qhznn" Nov 04 04:56:26.902217 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:26.902125 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-db8fae-856fc89fc8-x97fb"] Nov 04 04:56:26.906756 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:26.906728 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-db8fae-856fc89fc8-x97fb" Nov 04 04:56:26.948823 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:26.948800 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-db8fae-856fc89fc8-x97fb"] Nov 04 04:56:26.969900 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:26.969863 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btmvd\" (UniqueName: \"kubernetes.io/projected/24037ab8-fddf-4dd8-b170-96bdc4279e3f-kube-api-access-btmvd\") pod \"payload-processing-e2e-worker-w6-db8fae-856fc89fc8-x97fb\" (UID: \"24037ab8-fddf-4dd8-b170-96bdc4279e3f\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-db8fae-856fc89fc8-x97fb" Nov 04 04:56:26.970177 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:26.970155 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24037ab8-fddf-4dd8-b170-96bdc4279e3f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-db8fae-856fc89fc8-x97fb\" (UID: \"24037ab8-fddf-4dd8-b170-96bdc4279e3f\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-db8fae-856fc89fc8-x97fb" Nov 04 04:56:26.992818 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:26.992786 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-db8fae\"" Nov 04 04:56:26.992912 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:26.992864 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-db8fae-dockercfg-nsvbz\"" Nov 04 04:56:27.070922 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:27.070891 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btmvd\" (UniqueName: \"kubernetes.io/projected/24037ab8-fddf-4dd8-b170-96bdc4279e3f-kube-api-access-btmvd\") pod \"payload-processing-e2e-worker-w6-db8fae-856fc89fc8-x97fb\" (UID: \"24037ab8-fddf-4dd8-b170-96bdc4279e3f\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-db8fae-856fc89fc8-x97fb" Nov 04 04:56:27.071055 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:27.071011 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24037ab8-fddf-4dd8-b170-96bdc4279e3f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-db8fae-856fc89fc8-x97fb\" (UID: \"24037ab8-fddf-4dd8-b170-96bdc4279e3f\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-db8fae-856fc89fc8-x97fb" Nov 04 04:56:27.071708 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:27.071691 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24037ab8-fddf-4dd8-b170-96bdc4279e3f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-db8fae-856fc89fc8-x97fb\" (UID: \"24037ab8-fddf-4dd8-b170-96bdc4279e3f\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-db8fae-856fc89fc8-x97fb" Nov 04 04:56:27.128410 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:27.128378 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btmvd\" (UniqueName: \"kubernetes.io/projected/24037ab8-fddf-4dd8-b170-96bdc4279e3f-kube-api-access-btmvd\") pod \"payload-processing-e2e-worker-w6-db8fae-856fc89fc8-x97fb\" (UID: \"24037ab8-fddf-4dd8-b170-96bdc4279e3f\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-db8fae-856fc89fc8-x97fb" Nov 04 04:56:27.221287 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:27.221204 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-db8fae-856fc89fc8-x97fb" Nov 04 04:56:27.417429 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:27.417400 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 04:56:27.417636 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:27.417479 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" podUID="7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 04:56:27.609347 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:27.609147 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-db8fae-856fc89fc8-x97fb"] Nov 04 04:56:28.615724 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:28.615695 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-db8fae-856fc89fc8-x97fb" event={"ID":"24037ab8-fddf-4dd8-b170-96bdc4279e3f","Type":"ContainerStarted","Data":"106d718d23426bfbae521a1071b7b9bffd115cf74f6a1b71cae79af7f101192e"} Nov 04 04:56:28.616136 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:28.615754 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-db8fae-856fc89fc8-x97fb" event={"ID":"24037ab8-fddf-4dd8-b170-96bdc4279e3f","Type":"ContainerStarted","Data":"2e216106bafc503a128524e7ca87146a45ee7da856d75fe3cebf3896ee124ffa"} Nov 04 04:56:28.616136 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:28.615786 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-db8fae-856fc89fc8-x97fb" Nov 04 04:56:28.617248 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:28.617223 4395 generic.go:356] "Generic (PLEG): container finished" podID="7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3" containerID="96dc40bd86256d67a635da6966dd828d2c6f96b60a30da6fd1ff64b4a5e007a0" exitCode=0 Nov 04 04:56:28.617389 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:28.617322 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" event={"ID":"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3","Type":"ContainerDied","Data":"96dc40bd86256d67a635da6966dd828d2c6f96b60a30da6fd1ff64b4a5e007a0"} Nov 04 04:56:28.617662 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:28.617646 4395 scope.go:122] "RemoveContainer" containerID="96dc40bd86256d67a635da6966dd828d2c6f96b60a30da6fd1ff64b4a5e007a0" Nov 04 04:56:28.618818 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:28.618792 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 04:56:28.618899 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:28.618861 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 04:56:28.618899 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:28.618891 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 04:56:28.726134 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:28.726086 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-db8fae-856fc89fc8-x97fb" podStartSLOduration=2.726073005 podStartE2EDuration="2.726073005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:56:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:56:28.723471401 +0000 UTC m=+630.413553848" watchObservedRunningTime="2026-11-04 04:56:28.726073005 +0000 UTC m=+630.416155452" Nov 04 04:56:29.642312 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:29.642262 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" event={"ID":"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3","Type":"ContainerStarted","Data":"b891c8d18324b1bcf09778beb94367415dc1db0632fad3296f15789f7af7cc44"} Nov 04 04:56:30.201721 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:30.201687 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" Nov 04 04:56:30.202069 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:30.201987 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 04 04:56:30.202238 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:30.202214 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" podUID="7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 04 04:56:31.209321 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:31.208711 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 04 04:56:31.209321 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:31.208766 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" podUID="7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 04 04:56:31.552496 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:31.552472 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-19bf5b-7c6d784c4-s8rpn" Nov 04 04:56:32.201320 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:32.200651 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 04 04:56:32.201320 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:32.200724 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" podUID="7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 04 04:56:33.201323 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:33.200705 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 04 04:56:33.201323 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:33.200787 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" podUID="7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 04 04:56:33.506373 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:56:33.505932 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 04:56:34.205324 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:34.205238 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 04 04:56:34.205324 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:34.205314 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" podUID="7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 04 04:56:35.204731 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:35.204695 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" Nov 04 04:56:35.205088 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:35.205067 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" Nov 04 04:56:35.205877 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:35.205854 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" Nov 04 04:56:38.736411 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:38.736381 4395 generic.go:356] "Generic (PLEG): container finished" podID="5798fb27-7552-42f7-ae48-c4ff7decff83" containerID="f9e1e716d3e3904299a6cf769a4559d76a778f432282c57bea1b14556a26994b" exitCode=0 Nov 04 04:56:38.736838 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:38.736449 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" event={"ID":"5798fb27-7552-42f7-ae48-c4ff7decff83","Type":"ContainerDied","Data":"f9e1e716d3e3904299a6cf769a4559d76a778f432282c57bea1b14556a26994b"} Nov 04 04:56:38.736838 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:38.736830 4395 scope.go:122] "RemoveContainer" containerID="f9e1e716d3e3904299a6cf769a4559d76a778f432282c57bea1b14556a26994b" Nov 04 04:56:38.738152 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:38.738121 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 04:56:38.738222 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:38.738195 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 04:56:38.738256 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:38.738227 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 04:56:39.645610 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:39.645580 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-db8fae-856fc89fc8-x97fb" Nov 04 04:56:39.744848 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:39.744816 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" event={"ID":"5798fb27-7552-42f7-ae48-c4ff7decff83","Type":"ContainerStarted","Data":"f689453b02d1e5b4f217640b7db32540b25105755d4f44424aaa2bc083efe3d0"} Nov 04 04:56:40.607717 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:40.607681 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" Nov 04 04:56:40.608728 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:40.608702 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 04 04:56:40.608852 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:40.608784 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" podUID="5798fb27-7552-42f7-ae48-c4ff7decff83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 04 04:56:40.627312 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:40.627276 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv"] Nov 04 04:56:40.630223 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:40.630205 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv" Nov 04 04:56:40.649515 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:40.649496 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0574cb11-9f48-409a-8735-a4ab23411f4e-metrics-certs\") pod \"maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv\" (UID: \"0574cb11-9f48-409a-8735-a4ab23411f4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv" Nov 04 04:56:40.649616 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:40.649550 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpnt8\" (UniqueName: \"kubernetes.io/projected/0574cb11-9f48-409a-8735-a4ab23411f4e-kube-api-access-xpnt8\") pod \"maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv\" (UID: \"0574cb11-9f48-409a-8735-a4ab23411f4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv" Nov 04 04:56:40.649681 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:40.649661 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0574cb11-9f48-409a-8735-a4ab23411f4e-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv\" (UID: \"0574cb11-9f48-409a-8735-a4ab23411f4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv" Nov 04 04:56:40.649728 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:40.649699 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0574cb11-9f48-409a-8735-a4ab23411f4e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv\" (UID: \"0574cb11-9f48-409a-8735-a4ab23411f4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv" Nov 04 04:56:40.686554 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:40.686524 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-204fa0-serving-cert\"" Nov 04 04:56:40.686700 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:40.686633 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 04:56:40.686700 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:40.686693 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 04:56:40.707900 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:40.707875 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bv7vs\"" Nov 04 04:56:40.708041 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:40.707954 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 04 04:56:40.735125 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:40.735103 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv"] Nov 04 04:56:40.750259 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:40.750236 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0574cb11-9f48-409a-8735-a4ab23411f4e-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv\" (UID: \"0574cb11-9f48-409a-8735-a4ab23411f4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv" Nov 04 04:56:40.750610 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:40.750273 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0574cb11-9f48-409a-8735-a4ab23411f4e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv\" (UID: \"0574cb11-9f48-409a-8735-a4ab23411f4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv" Nov 04 04:56:40.750610 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:40.750359 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0574cb11-9f48-409a-8735-a4ab23411f4e-metrics-certs\") pod \"maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv\" (UID: \"0574cb11-9f48-409a-8735-a4ab23411f4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv" Nov 04 04:56:40.750610 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:40.750429 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpnt8\" (UniqueName: \"kubernetes.io/projected/0574cb11-9f48-409a-8735-a4ab23411f4e-kube-api-access-xpnt8\") pod \"maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv\" (UID: \"0574cb11-9f48-409a-8735-a4ab23411f4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv" Nov 04 04:56:40.750610 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:56:40.750480 4395 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-204fa0-serving-cert: secret "maas-api-e2e-worker-w0-204fa0-serving-cert" not found Nov 04 04:56:40.750610 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:56:40.750550 4395 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0574cb11-9f48-409a-8735-a4ab23411f4e-maas-api-tls podName:0574cb11-9f48-409a-8735-a4ab23411f4e nodeName:}" failed. No retries permitted until 2026-11-04 04:56:41.25053293 +0000 UTC m=+642.940615384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0574cb11-9f48-409a-8735-a4ab23411f4e-maas-api-tls") pod "maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv" (UID: "0574cb11-9f48-409a-8735-a4ab23411f4e") : secret "maas-api-e2e-worker-w0-204fa0-serving-cert" not found Nov 04 04:56:40.750991 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:40.750975 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0574cb11-9f48-409a-8735-a4ab23411f4e-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv\" (UID: \"0574cb11-9f48-409a-8735-a4ab23411f4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv" Nov 04 04:56:40.752785 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:40.752766 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0574cb11-9f48-409a-8735-a4ab23411f4e-metrics-certs\") pod \"maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv\" (UID: \"0574cb11-9f48-409a-8735-a4ab23411f4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv" Nov 04 04:56:40.811557 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:40.811530 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpnt8\" (UniqueName: \"kubernetes.io/projected/0574cb11-9f48-409a-8735-a4ab23411f4e-kube-api-access-xpnt8\") pod \"maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv\" (UID: \"0574cb11-9f48-409a-8735-a4ab23411f4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv" Nov 04 04:56:41.255886 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:41.255857 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0574cb11-9f48-409a-8735-a4ab23411f4e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv\" (UID: \"0574cb11-9f48-409a-8735-a4ab23411f4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv" Nov 04 04:56:41.265381 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:41.258730 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0574cb11-9f48-409a-8735-a4ab23411f4e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv\" (UID: \"0574cb11-9f48-409a-8735-a4ab23411f4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv" Nov 04 04:56:41.377467 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:41.377415 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" Nov 04 04:56:41.542514 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:41.542437 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv" Nov 04 04:56:41.606810 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:41.606782 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 04 04:56:41.606965 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:41.606831 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" podUID="5798fb27-7552-42f7-ae48-c4ff7decff83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 04 04:56:41.952932 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:41.952897 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv"] Nov 04 04:56:41.964715 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:56:41.960681 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0574cb11_9f48_409a_8735_a4ab23411f4e.slice/crio-b032d9b6beb1e3972c73a90cf5b0efcfb7d20f2d149fc6fb0aa2d8a24ffb95ae WatchSource:0}: Error finding container b032d9b6beb1e3972c73a90cf5b0efcfb7d20f2d149fc6fb0aa2d8a24ffb95ae: Status 404 returned error can't find the container with id b032d9b6beb1e3972c73a90cf5b0efcfb7d20f2d149fc6fb0aa2d8a24ffb95ae Nov 04 04:56:41.964715 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:41.961701 4395 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 04:56:42.607111 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:42.607075 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 04 04:56:42.607357 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:42.607130 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" podUID="5798fb27-7552-42f7-ae48-c4ff7decff83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 04 04:56:42.780422 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:42.780385 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv" event={"ID":"0574cb11-9f48-409a-8735-a4ab23411f4e","Type":"ContainerStarted","Data":"b032d9b6beb1e3972c73a90cf5b0efcfb7d20f2d149fc6fb0aa2d8a24ffb95ae"} Nov 04 04:56:43.607769 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:43.606845 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 04 04:56:43.607769 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:43.606901 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" podUID="5798fb27-7552-42f7-ae48-c4ff7decff83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 04 04:56:44.607319 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:44.607254 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 04 04:56:44.607510 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:44.607335 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" podUID="5798fb27-7552-42f7-ae48-c4ff7decff83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 04 04:56:44.803057 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:44.803019 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv" event={"ID":"0574cb11-9f48-409a-8735-a4ab23411f4e","Type":"ContainerStarted","Data":"83b76a5920157704124c731b4eeb584253178feef18f18a916cb37ff025d9dbf"} Nov 04 04:56:44.803516 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:44.803171 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv" Nov 04 04:56:44.931383 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:44.930685 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv" podStartSLOduration=2.20495681 podStartE2EDuration="4.93066708s" totalImagesPullingTime="2.72571027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:56:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:56:44.929600844 +0000 UTC m=+646.619683290" watchObservedRunningTime="2026-11-04 04:56:44.93066708 +0000 UTC m=+646.620749527" Nov 04 04:56:45.611922 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:45.611888 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" Nov 04 04:56:45.612896 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:45.612876 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" Nov 04 04:56:50.815022 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:50.814992 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv" Nov 04 04:56:54.865152 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:54.865099 4395 generic.go:356] "Generic (PLEG): container finished" podID="518e3ddc-9b6e-4127-a47c-397d9117a3ae" containerID="43742d519f2ebe3a7ef9a14ded98694cd286e629d0ed8159c3b8a4adf0cc81d4" exitCode=0 Nov 04 04:56:54.865507 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:54.865168 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" event={"ID":"518e3ddc-9b6e-4127-a47c-397d9117a3ae","Type":"ContainerDied","Data":"43742d519f2ebe3a7ef9a14ded98694cd286e629d0ed8159c3b8a4adf0cc81d4"} Nov 04 04:56:54.865553 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:54.865505 4395 scope.go:122] "RemoveContainer" containerID="43742d519f2ebe3a7ef9a14ded98694cd286e629d0ed8159c3b8a4adf0cc81d4" Nov 04 04:56:54.866755 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:54.866722 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 04:56:54.866824 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:54.866799 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 04:56:54.866861 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:54.866836 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 04:56:55.895320 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:55.889602 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" event={"ID":"518e3ddc-9b6e-4127-a47c-397d9117a3ae","Type":"ContainerStarted","Data":"f9f218bbf8a0b763a2ac2ef92fbb5325d8afbbe69a706e4b94d6c61c3a7cb527"} Nov 04 04:56:56.433962 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:56.433929 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" Nov 04 04:56:56.435064 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:56.435030 4395 patch_prober.go:29] interesting pod/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 04 04:56:56.435216 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:56.435077 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" podUID="518e3ddc-9b6e-4127-a47c-397d9117a3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 04 04:56:57.382626 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:57.382455 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" Nov 04 04:56:57.434479 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:57.434448 4395 patch_prober.go:29] interesting pod/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 04 04:56:57.434629 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:57.434511 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" podUID="518e3ddc-9b6e-4127-a47c-397d9117a3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 04 04:56:58.434062 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:58.434027 4395 patch_prober.go:29] interesting pod/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 04 04:56:58.434571 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:58.434100 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" podUID="518e3ddc-9b6e-4127-a47c-397d9117a3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 04 04:56:59.434435 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:59.434393 4395 patch_prober.go:29] interesting pod/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 04 04:56:59.434854 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:56:59.434454 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" podUID="518e3ddc-9b6e-4127-a47c-397d9117a3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 04 04:57:00.434479 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:00.434442 4395 patch_prober.go:29] interesting pod/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 04 04:57:00.434924 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:00.434506 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" podUID="518e3ddc-9b6e-4127-a47c-397d9117a3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 04 04:57:01.438709 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:01.438681 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" Nov 04 04:57:01.439744 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:01.439727 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" Nov 04 04:57:03.481530 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:57:03.481489 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 04:57:07.983167 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:07.983132 4395 generic.go:356] "Generic (PLEG): container finished" podID="24ea2807-ab22-495c-b66a-b35111e36b97" containerID="fb40a3de103e8fda221f07cce71ea4956f46b331866584e14b44b2409a8dc3d0" exitCode=0 Nov 04 04:57:07.983633 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:07.983175 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" event={"ID":"24ea2807-ab22-495c-b66a-b35111e36b97","Type":"ContainerDied","Data":"fb40a3de103e8fda221f07cce71ea4956f46b331866584e14b44b2409a8dc3d0"} Nov 04 04:57:07.983633 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:07.983520 4395 scope.go:122] "RemoveContainer" containerID="fb40a3de103e8fda221f07cce71ea4956f46b331866584e14b44b2409a8dc3d0" Nov 04 04:57:07.984781 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:07.984748 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 04:57:07.984873 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:07.984819 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 04:57:07.984873 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:07.984852 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 04:57:08.992394 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:08.992291 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" event={"ID":"24ea2807-ab22-495c-b66a-b35111e36b97","Type":"ContainerStarted","Data":"4eb04862a55327c22d4d40edd6f67fcb5868355224af7e0d5663fe70bba37ddf"} Nov 04 04:57:09.143346 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:09.143287 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" Nov 04 04:57:09.144482 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:09.144454 4395 patch_prober.go:29] interesting pod/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 04 04:57:09.144614 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:09.144503 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" podUID="24ea2807-ab22-495c-b66a-b35111e36b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 04 04:57:10.144404 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:10.144365 4395 patch_prober.go:29] interesting pod/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 04 04:57:10.144804 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:10.144440 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" podUID="24ea2807-ab22-495c-b66a-b35111e36b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 04 04:57:11.144006 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:11.143968 4395 patch_prober.go:29] interesting pod/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 04 04:57:11.144326 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:11.144273 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" podUID="24ea2807-ab22-495c-b66a-b35111e36b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 04 04:57:12.143609 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:12.143574 4395 patch_prober.go:29] interesting pod/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 04 04:57:12.144002 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:12.143625 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" podUID="24ea2807-ab22-495c-b66a-b35111e36b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 04 04:57:12.381401 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:12.381370 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" Nov 04 04:57:13.143742 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:13.143711 4395 patch_prober.go:29] interesting pod/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 04 04:57:13.144122 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:13.143757 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" podUID="24ea2807-ab22-495c-b66a-b35111e36b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 04 04:57:14.148334 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:14.148281 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" Nov 04 04:57:14.149338 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:14.149293 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" Nov 04 04:57:33.458764 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:57:33.458725 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 04:57:42.109241 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:42.109210 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-19bf5b/simulator-w5-19bf5b-kserve-58fb87c755-dqlhm"] Nov 04 04:57:42.112117 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:42.112088 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-19bf5b/simulator-w5-19bf5b-kserve-58fb87c755-dqlhm" Nov 04 04:57:42.141109 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:42.141081 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e577d1b4-d707-40c7-a78f-c64c480c1af2-home\") pod \"simulator-w5-19bf5b-kserve-58fb87c755-dqlhm\" (UID: \"e577d1b4-d707-40c7-a78f-c64c480c1af2\") " pod="e2e-models-e2e-worker-w5-19bf5b/simulator-w5-19bf5b-kserve-58fb87c755-dqlhm" Nov 04 04:57:42.141227 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:42.141133 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e577d1b4-d707-40c7-a78f-c64c480c1af2-tls-certs\") pod \"simulator-w5-19bf5b-kserve-58fb87c755-dqlhm\" (UID: \"e577d1b4-d707-40c7-a78f-c64c480c1af2\") " pod="e2e-models-e2e-worker-w5-19bf5b/simulator-w5-19bf5b-kserve-58fb87c755-dqlhm" Nov 04 04:57:42.141227 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:42.141195 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e577d1b4-d707-40c7-a78f-c64c480c1af2-model-cache\") pod \"simulator-w5-19bf5b-kserve-58fb87c755-dqlhm\" (UID: \"e577d1b4-d707-40c7-a78f-c64c480c1af2\") " pod="e2e-models-e2e-worker-w5-19bf5b/simulator-w5-19bf5b-kserve-58fb87c755-dqlhm" Nov 04 04:57:42.141325 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:42.141225 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czlj2\" (UniqueName: \"kubernetes.io/projected/e577d1b4-d707-40c7-a78f-c64c480c1af2-kube-api-access-czlj2\") pod \"simulator-w5-19bf5b-kserve-58fb87c755-dqlhm\" (UID: \"e577d1b4-d707-40c7-a78f-c64c480c1af2\") " pod="e2e-models-e2e-worker-w5-19bf5b/simulator-w5-19bf5b-kserve-58fb87c755-dqlhm" Nov 04 04:57:42.141325 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:42.141248 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e577d1b4-d707-40c7-a78f-c64c480c1af2-tmp-dir\") pod \"simulator-w5-19bf5b-kserve-58fb87c755-dqlhm\" (UID: \"e577d1b4-d707-40c7-a78f-c64c480c1af2\") " pod="e2e-models-e2e-worker-w5-19bf5b/simulator-w5-19bf5b-kserve-58fb87c755-dqlhm" Nov 04 04:57:42.141325 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:42.141279 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e577d1b4-d707-40c7-a78f-c64c480c1af2-dshm\") pod \"simulator-w5-19bf5b-kserve-58fb87c755-dqlhm\" (UID: \"e577d1b4-d707-40c7-a78f-c64c480c1af2\") " pod="e2e-models-e2e-worker-w5-19bf5b/simulator-w5-19bf5b-kserve-58fb87c755-dqlhm" Nov 04 04:57:42.160984 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:42.160961 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-19bf5b\"/\"openshift-service-ca.crt\"" Nov 04 04:57:42.207253 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:42.207218 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-19bf5b\"/\"kube-root-ca.crt\"" Nov 04 04:57:42.207412 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:42.207280 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-19bf5b\"/\"simulator-w5-19bf5b-kserve-self-signed-certs\"" Nov 04 04:57:42.207412 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:42.207321 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-19bf5b\"/\"default-dockercfg-mmfbl\"" Nov 04 04:57:42.209815 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:42.209792 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-19bf5b/simulator-w5-19bf5b-kserve-58fb87c755-dqlhm"] Nov 04 04:57:42.241796 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:42.241775 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e577d1b4-d707-40c7-a78f-c64c480c1af2-home\") pod \"simulator-w5-19bf5b-kserve-58fb87c755-dqlhm\" (UID: \"e577d1b4-d707-40c7-a78f-c64c480c1af2\") " pod="e2e-models-e2e-worker-w5-19bf5b/simulator-w5-19bf5b-kserve-58fb87c755-dqlhm" Nov 04 04:57:42.241898 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:42.241816 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e577d1b4-d707-40c7-a78f-c64c480c1af2-tls-certs\") pod \"simulator-w5-19bf5b-kserve-58fb87c755-dqlhm\" (UID: \"e577d1b4-d707-40c7-a78f-c64c480c1af2\") " pod="e2e-models-e2e-worker-w5-19bf5b/simulator-w5-19bf5b-kserve-58fb87c755-dqlhm" Nov 04 04:57:42.241898 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:42.241837 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e577d1b4-d707-40c7-a78f-c64c480c1af2-model-cache\") pod \"simulator-w5-19bf5b-kserve-58fb87c755-dqlhm\" (UID: \"e577d1b4-d707-40c7-a78f-c64c480c1af2\") " pod="e2e-models-e2e-worker-w5-19bf5b/simulator-w5-19bf5b-kserve-58fb87c755-dqlhm" Nov 04 04:57:42.241898 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:42.241860 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czlj2\" (UniqueName: \"kubernetes.io/projected/e577d1b4-d707-40c7-a78f-c64c480c1af2-kube-api-access-czlj2\") pod \"simulator-w5-19bf5b-kserve-58fb87c755-dqlhm\" (UID: \"e577d1b4-d707-40c7-a78f-c64c480c1af2\") " pod="e2e-models-e2e-worker-w5-19bf5b/simulator-w5-19bf5b-kserve-58fb87c755-dqlhm" Nov 04 04:57:42.241898 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:42.241879 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e577d1b4-d707-40c7-a78f-c64c480c1af2-tmp-dir\") pod \"simulator-w5-19bf5b-kserve-58fb87c755-dqlhm\" (UID: \"e577d1b4-d707-40c7-a78f-c64c480c1af2\") " pod="e2e-models-e2e-worker-w5-19bf5b/simulator-w5-19bf5b-kserve-58fb87c755-dqlhm" Nov 04 04:57:42.242087 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:42.241905 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e577d1b4-d707-40c7-a78f-c64c480c1af2-dshm\") pod \"simulator-w5-19bf5b-kserve-58fb87c755-dqlhm\" (UID: \"e577d1b4-d707-40c7-a78f-c64c480c1af2\") " pod="e2e-models-e2e-worker-w5-19bf5b/simulator-w5-19bf5b-kserve-58fb87c755-dqlhm" Nov 04 04:57:42.242251 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:42.242231 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e577d1b4-d707-40c7-a78f-c64c480c1af2-home\") pod \"simulator-w5-19bf5b-kserve-58fb87c755-dqlhm\" (UID: \"e577d1b4-d707-40c7-a78f-c64c480c1af2\") " pod="e2e-models-e2e-worker-w5-19bf5b/simulator-w5-19bf5b-kserve-58fb87c755-dqlhm" Nov 04 04:57:42.242344 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:42.242275 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e577d1b4-d707-40c7-a78f-c64c480c1af2-model-cache\") pod \"simulator-w5-19bf5b-kserve-58fb87c755-dqlhm\" (UID: \"e577d1b4-d707-40c7-a78f-c64c480c1af2\") " pod="e2e-models-e2e-worker-w5-19bf5b/simulator-w5-19bf5b-kserve-58fb87c755-dqlhm" Nov 04 04:57:42.242408 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:42.242361 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e577d1b4-d707-40c7-a78f-c64c480c1af2-tmp-dir\") pod \"simulator-w5-19bf5b-kserve-58fb87c755-dqlhm\" (UID: \"e577d1b4-d707-40c7-a78f-c64c480c1af2\") " pod="e2e-models-e2e-worker-w5-19bf5b/simulator-w5-19bf5b-kserve-58fb87c755-dqlhm" Nov 04 04:57:42.244110 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:42.244087 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e577d1b4-d707-40c7-a78f-c64c480c1af2-dshm\") pod \"simulator-w5-19bf5b-kserve-58fb87c755-dqlhm\" (UID: \"e577d1b4-d707-40c7-a78f-c64c480c1af2\") " pod="e2e-models-e2e-worker-w5-19bf5b/simulator-w5-19bf5b-kserve-58fb87c755-dqlhm" Nov 04 04:57:42.244284 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:42.244266 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e577d1b4-d707-40c7-a78f-c64c480c1af2-tls-certs\") pod \"simulator-w5-19bf5b-kserve-58fb87c755-dqlhm\" (UID: \"e577d1b4-d707-40c7-a78f-c64c480c1af2\") " pod="e2e-models-e2e-worker-w5-19bf5b/simulator-w5-19bf5b-kserve-58fb87c755-dqlhm" Nov 04 04:57:42.299565 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:42.299536 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czlj2\" (UniqueName: \"kubernetes.io/projected/e577d1b4-d707-40c7-a78f-c64c480c1af2-kube-api-access-czlj2\") pod \"simulator-w5-19bf5b-kserve-58fb87c755-dqlhm\" (UID: \"e577d1b4-d707-40c7-a78f-c64c480c1af2\") " pod="e2e-models-e2e-worker-w5-19bf5b/simulator-w5-19bf5b-kserve-58fb87c755-dqlhm" Nov 04 04:57:42.423281 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:42.423206 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-19bf5b/simulator-w5-19bf5b-kserve-58fb87c755-dqlhm" Nov 04 04:57:42.802911 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:42.802884 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-19bf5b/simulator-w5-19bf5b-kserve-58fb87c755-dqlhm"] Nov 04 04:57:42.805153 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:57:42.805119 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode577d1b4_d707_40c7_a78f_c64c480c1af2.slice/crio-38fdf2301d66b04b7508fab27336c7dd0720564755c6882bcd21993c1a027913 WatchSource:0}: Error finding container 38fdf2301d66b04b7508fab27336c7dd0720564755c6882bcd21993c1a027913: Status 404 returned error can't find the container with id 38fdf2301d66b04b7508fab27336c7dd0720564755c6882bcd21993c1a027913 Nov 04 04:57:43.186337 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:43.186225 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-19bf5b/simulator-w5-19bf5b-kserve-58fb87c755-dqlhm" event={"ID":"e577d1b4-d707-40c7-a78f-c64c480c1af2","Type":"ContainerStarted","Data":"857cd29c43cc69cd1b92e425e5bb73cbcbc000e4684fb48719cdebab93601af6"} Nov 04 04:57:43.186337 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:43.186278 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-19bf5b/simulator-w5-19bf5b-kserve-58fb87c755-dqlhm" event={"ID":"e577d1b4-d707-40c7-a78f-c64c480c1af2","Type":"ContainerStarted","Data":"38fdf2301d66b04b7508fab27336c7dd0720564755c6882bcd21993c1a027913"} Nov 04 04:57:43.296400 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:43.296354 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-19bf5b/simulator-w5-19bf5b-kserve-58fb87c755-dqlhm" podStartSLOduration=1.296339929 podStartE2EDuration="1.296339929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:57:43.294886791 +0000 UTC m=+704.984969237" watchObservedRunningTime="2026-11-04 04:57:43.296339929 +0000 UTC m=+704.986422376" Nov 04 04:57:43.424460 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:43.424378 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-19bf5b/simulator-w5-19bf5b-kserve-58fb87c755-dqlhm" Nov 04 04:57:52.424248 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:52.424217 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-19bf5b/simulator-w5-19bf5b-kserve-58fb87c755-dqlhm" Nov 04 04:57:52.437067 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:52.437043 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-19bf5b/simulator-w5-19bf5b-kserve-58fb87c755-dqlhm" Nov 04 04:57:53.254337 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:53.254290 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-19bf5b/simulator-w5-19bf5b-kserve-58fb87c755-dqlhm" Nov 04 04:57:54.667867 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:54.667833 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r"] Nov 04 04:57:54.671125 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:54.671102 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" Nov 04 04:57:54.719918 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:54.719884 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-c5bb6d-openshift-default-dockercfg-7x2zp\"" Nov 04 04:57:54.763204 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:54.763173 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-istio-data\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r\" (UID: \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" Nov 04 04:57:54.763372 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:54.763216 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-workload-certs\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r\" (UID: \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" Nov 04 04:57:54.763372 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:54.763246 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d58pz\" (UniqueName: \"kubernetes.io/projected/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-kube-api-access-d58pz\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r\" (UID: \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" Nov 04 04:57:54.763483 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:54.763393 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-istio-token\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r\" (UID: \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" Nov 04 04:57:54.763483 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:54.763454 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-workload-socket\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r\" (UID: \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" Nov 04 04:57:54.763483 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:54.763477 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-istio-envoy\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r\" (UID: \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" Nov 04 04:57:54.763632 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:54.763507 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-istio-podinfo\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r\" (UID: \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" Nov 04 04:57:54.763686 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:54.763643 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-credential-socket\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r\" (UID: \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" Nov 04 04:57:54.763732 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:54.763717 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-istiod-ca-cert\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r\" (UID: \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" Nov 04 04:57:54.775439 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:54.775413 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r"] Nov 04 04:57:54.864783 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:54.864751 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-istiod-ca-cert\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r\" (UID: \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" Nov 04 04:57:54.864958 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:54.864803 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-istio-data\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r\" (UID: \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" Nov 04 04:57:54.864958 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:54.864917 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-workload-certs\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r\" (UID: \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" Nov 04 04:57:54.865045 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:54.864981 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d58pz\" (UniqueName: \"kubernetes.io/projected/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-kube-api-access-d58pz\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r\" (UID: \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" Nov 04 04:57:54.865045 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:54.865009 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-istio-token\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r\" (UID: \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" Nov 04 04:57:54.865139 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:54.865049 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-workload-socket\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r\" (UID: \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" Nov 04 04:57:54.865206 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:54.865186 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-istio-data\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r\" (UID: \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" Nov 04 04:57:54.865265 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:54.865204 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-istio-envoy\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r\" (UID: \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" Nov 04 04:57:54.865265 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:54.865245 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-istio-podinfo\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r\" (UID: \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" Nov 04 04:57:54.865421 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:54.865340 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-workload-certs\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r\" (UID: \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" Nov 04 04:57:54.865478 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:54.865415 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-workload-socket\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r\" (UID: \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" Nov 04 04:57:54.865564 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:54.865541 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-credential-socket\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r\" (UID: \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" Nov 04 04:57:54.865633 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:54.865582 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-istiod-ca-cert\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r\" (UID: \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" Nov 04 04:57:54.865814 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:54.865797 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-credential-socket\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r\" (UID: \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" Nov 04 04:57:54.867537 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:54.867512 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-istio-envoy\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r\" (UID: \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" Nov 04 04:57:54.868066 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:54.868044 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-istio-podinfo\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r\" (UID: \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" Nov 04 04:57:54.926135 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:54.926071 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d58pz\" (UniqueName: \"kubernetes.io/projected/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-kube-api-access-d58pz\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r\" (UID: \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" Nov 04 04:57:54.928837 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:54.928817 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-istio-token\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r\" (UID: \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" Nov 04 04:57:54.986982 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:54.986948 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" Nov 04 04:57:55.421870 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:55.421842 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r"] Nov 04 04:57:55.423741 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:57:55.423713 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11dc70f6_5166_4ec0_a746_9d75ee2c6e56.slice/crio-43f72e8956bf6fb36559366274e8cc5457c82d92447053ae52b5d56e5e126b81 WatchSource:0}: Error finding container 43f72e8956bf6fb36559366274e8cc5457c82d92447053ae52b5d56e5e126b81: Status 404 returned error can't find the container with id 43f72e8956bf6fb36559366274e8cc5457c82d92447053ae52b5d56e5e126b81 Nov 04 04:57:55.425904 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:55.425861 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 04:57:55.425989 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:55.425946 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 04:57:55.425989 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:55.425975 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 04:57:56.286815 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:56.286776 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" event={"ID":"11dc70f6-5166-4ec0-a746-9d75ee2c6e56","Type":"ContainerStarted","Data":"1fc2684f299796b9b55ae5a33809495dfe29551cd03fcd9f67195045d9e76417"} Nov 04 04:57:56.287367 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:56.286829 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" event={"ID":"11dc70f6-5166-4ec0-a746-9d75ee2c6e56","Type":"ContainerStarted","Data":"43f72e8956bf6fb36559366274e8cc5457c82d92447053ae52b5d56e5e126b81"} Nov 04 04:57:56.407475 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:56.407426 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" podStartSLOduration=2.407412478 podStartE2EDuration="2.407412478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:57:56.404874199 +0000 UTC m=+718.094956646" watchObservedRunningTime="2026-11-04 04:57:56.407412478 +0000 UTC m=+718.097494955" Nov 04 04:57:56.987394 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:56.987355 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" Nov 04 04:57:56.989213 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:56.989187 4395 patch_prober.go:29] interesting pod/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 04 04:57:56.989361 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:56.989237 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" podUID="11dc70f6-5166-4ec0-a746-9d75ee2c6e56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 04 04:57:57.987667 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:57.987622 4395 patch_prober.go:29] interesting pod/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 04 04:57:57.988184 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:57.987677 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" podUID="11dc70f6-5166-4ec0-a746-9d75ee2c6e56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 04 04:57:58.988738 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:58.988698 4395 patch_prober.go:29] interesting pod/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 04 04:57:58.989390 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:58.989355 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" podUID="11dc70f6-5166-4ec0-a746-9d75ee2c6e56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 04 04:57:59.988094 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:59.988064 4395 patch_prober.go:29] interesting pod/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 04 04:57:59.988266 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:57:59.988124 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" podUID="11dc70f6-5166-4ec0-a746-9d75ee2c6e56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 04 04:58:00.988066 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:58:00.988031 4395 patch_prober.go:29] interesting pod/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 04 04:58:00.988485 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:58:00.988084 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" podUID="11dc70f6-5166-4ec0-a746-9d75ee2c6e56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 04 04:58:02.286645 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:58:02.286607 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" Nov 04 04:58:02.287470 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:58:02.287451 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" Nov 04 04:58:02.288563 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:58:02.288532 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" Nov 04 04:58:03.457805 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:58:03.457765 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 04:58:08.588258 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:58:08.588223 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-204fa0/simulator-w0-204fa0-kserve-64b5544888-df4pp"] Nov 04 04:58:08.591726 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:58:08.591702 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-204fa0/simulator-w0-204fa0-kserve-64b5544888-df4pp" Nov 04 04:58:08.686836 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:58:08.686810 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-204fa0\"/\"kube-root-ca.crt\"" Nov 04 04:58:08.686836 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:58:08.686812 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-204fa0\"/\"openshift-service-ca.crt\"" Nov 04 04:58:08.686836 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:58:08.686818 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-204fa0\"/\"default-dockercfg-g5fpr\"" Nov 04 04:58:08.687055 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:58:08.686896 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-204fa0\"/\"simulator-w0-204fa0-kserve-self-signed-certs\"" Nov 04 04:58:08.689491 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:58:08.689466 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-204fa0/simulator-w0-204fa0-kserve-64b5544888-df4pp"] Nov 04 04:58:08.740485 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:58:08.740442 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80a73743-0113-4f1c-94cc-38b67249b698-tmp-dir\") pod \"simulator-w0-204fa0-kserve-64b5544888-df4pp\" (UID: \"80a73743-0113-4f1c-94cc-38b67249b698\") " pod="e2e-models-e2e-worker-w0-204fa0/simulator-w0-204fa0-kserve-64b5544888-df4pp" Nov 04 04:58:08.740823 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:58:08.740801 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfts2\" (UniqueName: \"kubernetes.io/projected/80a73743-0113-4f1c-94cc-38b67249b698-kube-api-access-tfts2\") pod \"simulator-w0-204fa0-kserve-64b5544888-df4pp\" (UID: \"80a73743-0113-4f1c-94cc-38b67249b698\") " pod="e2e-models-e2e-worker-w0-204fa0/simulator-w0-204fa0-kserve-64b5544888-df4pp" Nov 04 04:58:08.740976 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:58:08.740954 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80a73743-0113-4f1c-94cc-38b67249b698-home\") pod \"simulator-w0-204fa0-kserve-64b5544888-df4pp\" (UID: \"80a73743-0113-4f1c-94cc-38b67249b698\") " pod="e2e-models-e2e-worker-w0-204fa0/simulator-w0-204fa0-kserve-64b5544888-df4pp" Nov 04 04:58:08.741131 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:58:08.741021 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80a73743-0113-4f1c-94cc-38b67249b698-dshm\") pod \"simulator-w0-204fa0-kserve-64b5544888-df4pp\" (UID: \"80a73743-0113-4f1c-94cc-38b67249b698\") " pod="e2e-models-e2e-worker-w0-204fa0/simulator-w0-204fa0-kserve-64b5544888-df4pp" Nov 04 04:58:08.741131 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:58:08.741049 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80a73743-0113-4f1c-94cc-38b67249b698-model-cache\") pod \"simulator-w0-204fa0-kserve-64b5544888-df4pp\" (UID: \"80a73743-0113-4f1c-94cc-38b67249b698\") " pod="e2e-models-e2e-worker-w0-204fa0/simulator-w0-204fa0-kserve-64b5544888-df4pp" Nov 04 04:58:08.741251 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:58:08.741141 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80a73743-0113-4f1c-94cc-38b67249b698-tls-certs\") pod \"simulator-w0-204fa0-kserve-64b5544888-df4pp\" (UID: \"80a73743-0113-4f1c-94cc-38b67249b698\") " pod="e2e-models-e2e-worker-w0-204fa0/simulator-w0-204fa0-kserve-64b5544888-df4pp" Nov 04 04:58:08.841730 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:58:08.841647 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfts2\" (UniqueName: \"kubernetes.io/projected/80a73743-0113-4f1c-94cc-38b67249b698-kube-api-access-tfts2\") pod \"simulator-w0-204fa0-kserve-64b5544888-df4pp\" (UID: \"80a73743-0113-4f1c-94cc-38b67249b698\") " pod="e2e-models-e2e-worker-w0-204fa0/simulator-w0-204fa0-kserve-64b5544888-df4pp" Nov 04 04:58:08.841730 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:58:08.841698 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80a73743-0113-4f1c-94cc-38b67249b698-home\") pod \"simulator-w0-204fa0-kserve-64b5544888-df4pp\" (UID: \"80a73743-0113-4f1c-94cc-38b67249b698\") " pod="e2e-models-e2e-worker-w0-204fa0/simulator-w0-204fa0-kserve-64b5544888-df4pp" Nov 04 04:58:08.841925 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:58:08.841735 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80a73743-0113-4f1c-94cc-38b67249b698-dshm\") pod \"simulator-w0-204fa0-kserve-64b5544888-df4pp\" (UID: \"80a73743-0113-4f1c-94cc-38b67249b698\") " pod="e2e-models-e2e-worker-w0-204fa0/simulator-w0-204fa0-kserve-64b5544888-df4pp" Nov 04 04:58:08.841925 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:58:08.841752 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80a73743-0113-4f1c-94cc-38b67249b698-model-cache\") pod \"simulator-w0-204fa0-kserve-64b5544888-df4pp\" (UID: \"80a73743-0113-4f1c-94cc-38b67249b698\") " pod="e2e-models-e2e-worker-w0-204fa0/simulator-w0-204fa0-kserve-64b5544888-df4pp" Nov 04 04:58:08.841925 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:58:08.841811 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80a73743-0113-4f1c-94cc-38b67249b698-tls-certs\") pod \"simulator-w0-204fa0-kserve-64b5544888-df4pp\" (UID: \"80a73743-0113-4f1c-94cc-38b67249b698\") " pod="e2e-models-e2e-worker-w0-204fa0/simulator-w0-204fa0-kserve-64b5544888-df4pp" Nov 04 04:58:08.841925 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:58:08.841850 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80a73743-0113-4f1c-94cc-38b67249b698-tmp-dir\") pod \"simulator-w0-204fa0-kserve-64b5544888-df4pp\" (UID: \"80a73743-0113-4f1c-94cc-38b67249b698\") " pod="e2e-models-e2e-worker-w0-204fa0/simulator-w0-204fa0-kserve-64b5544888-df4pp" Nov 04 04:58:08.842271 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:58:08.842242 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80a73743-0113-4f1c-94cc-38b67249b698-home\") pod \"simulator-w0-204fa0-kserve-64b5544888-df4pp\" (UID: \"80a73743-0113-4f1c-94cc-38b67249b698\") " pod="e2e-models-e2e-worker-w0-204fa0/simulator-w0-204fa0-kserve-64b5544888-df4pp" Nov 04 04:58:08.842442 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:58:08.842275 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80a73743-0113-4f1c-94cc-38b67249b698-tmp-dir\") pod \"simulator-w0-204fa0-kserve-64b5544888-df4pp\" (UID: \"80a73743-0113-4f1c-94cc-38b67249b698\") " pod="e2e-models-e2e-worker-w0-204fa0/simulator-w0-204fa0-kserve-64b5544888-df4pp" Nov 04 04:58:08.842442 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:58:08.842330 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80a73743-0113-4f1c-94cc-38b67249b698-model-cache\") pod \"simulator-w0-204fa0-kserve-64b5544888-df4pp\" (UID: \"80a73743-0113-4f1c-94cc-38b67249b698\") " pod="e2e-models-e2e-worker-w0-204fa0/simulator-w0-204fa0-kserve-64b5544888-df4pp" Nov 04 04:58:08.844154 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:58:08.844123 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80a73743-0113-4f1c-94cc-38b67249b698-dshm\") pod \"simulator-w0-204fa0-kserve-64b5544888-df4pp\" (UID: \"80a73743-0113-4f1c-94cc-38b67249b698\") " pod="e2e-models-e2e-worker-w0-204fa0/simulator-w0-204fa0-kserve-64b5544888-df4pp" Nov 04 04:58:08.844438 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:58:08.844421 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80a73743-0113-4f1c-94cc-38b67249b698-tls-certs\") pod \"simulator-w0-204fa0-kserve-64b5544888-df4pp\" (UID: \"80a73743-0113-4f1c-94cc-38b67249b698\") " pod="e2e-models-e2e-worker-w0-204fa0/simulator-w0-204fa0-kserve-64b5544888-df4pp" Nov 04 04:58:08.898956 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:58:08.898925 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfts2\" (UniqueName: \"kubernetes.io/projected/80a73743-0113-4f1c-94cc-38b67249b698-kube-api-access-tfts2\") pod \"simulator-w0-204fa0-kserve-64b5544888-df4pp\" (UID: \"80a73743-0113-4f1c-94cc-38b67249b698\") " pod="e2e-models-e2e-worker-w0-204fa0/simulator-w0-204fa0-kserve-64b5544888-df4pp" Nov 04 04:58:08.902766 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:58:08.902744 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-204fa0/simulator-w0-204fa0-kserve-64b5544888-df4pp" Nov 04 04:58:09.277478 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:58:09.277453 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-204fa0/simulator-w0-204fa0-kserve-64b5544888-df4pp"] Nov 04 04:58:09.279219 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:58:09.279177 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80a73743_0113_4f1c_94cc_38b67249b698.slice/crio-48c812fb6f66537c00cd572c18202f5c8cb8c842b2a0437c08218289c1a117d9 WatchSource:0}: Error finding container 48c812fb6f66537c00cd572c18202f5c8cb8c842b2a0437c08218289c1a117d9: Status 404 returned error can't find the container with id 48c812fb6f66537c00cd572c18202f5c8cb8c842b2a0437c08218289c1a117d9 Nov 04 04:58:09.370582 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:58:09.370555 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-204fa0/simulator-w0-204fa0-kserve-64b5544888-df4pp" event={"ID":"80a73743-0113-4f1c-94cc-38b67249b698","Type":"ContainerStarted","Data":"7bc11ae9ec0cd0b2bcc5789c5c3470f5c1d11553ec6fdb3fc1f9d80017373fbc"} Nov 04 04:58:09.370708 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:58:09.370595 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-204fa0/simulator-w0-204fa0-kserve-64b5544888-df4pp" event={"ID":"80a73743-0113-4f1c-94cc-38b67249b698","Type":"ContainerStarted","Data":"48c812fb6f66537c00cd572c18202f5c8cb8c842b2a0437c08218289c1a117d9"} Nov 04 04:58:09.479695 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:58:09.479653 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-204fa0/simulator-w0-204fa0-kserve-64b5544888-df4pp" podStartSLOduration=1.479640436 podStartE2EDuration="1.479640436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:58:09.478738124 +0000 UTC m=+731.168820571" watchObservedRunningTime="2026-11-04 04:58:09.479640436 +0000 UTC m=+731.169722883" Nov 04 04:58:09.903244 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:58:09.903218 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-204fa0/simulator-w0-204fa0-kserve-64b5544888-df4pp" Nov 04 04:58:18.903229 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:58:18.903196 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-204fa0/simulator-w0-204fa0-kserve-64b5544888-df4pp" Nov 04 04:58:18.916065 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:58:18.916043 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-204fa0/simulator-w0-204fa0-kserve-64b5544888-df4pp" Nov 04 04:58:19.442381 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:58:19.442351 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-204fa0/simulator-w0-204fa0-kserve-64b5544888-df4pp" Nov 04 04:58:33.456963 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:58:33.456928 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 04:59:03.457275 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:59:03.457238 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 04:59:12.992364 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:12.992329 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-19bf5b/premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b"] Nov 04 04:59:13.000647 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:13.000070 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-19bf5b/premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b" Nov 04 04:59:13.052984 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:13.052950 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-19bf5b/premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b"] Nov 04 04:59:13.095979 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:13.095941 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-19bf5b\"/\"premium-w5-19bf5b-kserve-self-signed-certs\"" Nov 04 04:59:13.101645 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:13.101619 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-tmp-dir\") pod \"premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b\" (UID: \"ac06b52c-cb97-479d-813f-a9e9d6d9e67a\") " pod="e2e-models-e2e-worker-w5-19bf5b/premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b" Nov 04 04:59:13.101792 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:13.101658 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-dshm\") pod \"premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b\" (UID: \"ac06b52c-cb97-479d-813f-a9e9d6d9e67a\") " pod="e2e-models-e2e-worker-w5-19bf5b/premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b" Nov 04 04:59:13.101792 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:13.101675 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-model-cache\") pod \"premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b\" (UID: \"ac06b52c-cb97-479d-813f-a9e9d6d9e67a\") " pod="e2e-models-e2e-worker-w5-19bf5b/premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b" Nov 04 04:59:13.101921 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:13.101791 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-tls-certs\") pod \"premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b\" (UID: \"ac06b52c-cb97-479d-813f-a9e9d6d9e67a\") " pod="e2e-models-e2e-worker-w5-19bf5b/premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b" Nov 04 04:59:13.101921 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:13.101819 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrq64\" (UniqueName: \"kubernetes.io/projected/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-kube-api-access-zrq64\") pod \"premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b\" (UID: \"ac06b52c-cb97-479d-813f-a9e9d6d9e67a\") " pod="e2e-models-e2e-worker-w5-19bf5b/premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b" Nov 04 04:59:13.101921 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:13.101848 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-home\") pod \"premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b\" (UID: \"ac06b52c-cb97-479d-813f-a9e9d6d9e67a\") " pod="e2e-models-e2e-worker-w5-19bf5b/premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b" Nov 04 04:59:13.202610 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:13.202576 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-tmp-dir\") pod \"premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b\" (UID: \"ac06b52c-cb97-479d-813f-a9e9d6d9e67a\") " pod="e2e-models-e2e-worker-w5-19bf5b/premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b" Nov 04 04:59:13.202790 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:13.202641 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-dshm\") pod \"premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b\" (UID: \"ac06b52c-cb97-479d-813f-a9e9d6d9e67a\") " pod="e2e-models-e2e-worker-w5-19bf5b/premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b" Nov 04 04:59:13.202790 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:13.202671 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-model-cache\") pod \"premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b\" (UID: \"ac06b52c-cb97-479d-813f-a9e9d6d9e67a\") " pod="e2e-models-e2e-worker-w5-19bf5b/premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b" Nov 04 04:59:13.202790 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:13.202716 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-tls-certs\") pod \"premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b\" (UID: \"ac06b52c-cb97-479d-813f-a9e9d6d9e67a\") " pod="e2e-models-e2e-worker-w5-19bf5b/premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b" Nov 04 04:59:13.202790 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:13.202741 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrq64\" (UniqueName: \"kubernetes.io/projected/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-kube-api-access-zrq64\") pod \"premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b\" (UID: \"ac06b52c-cb97-479d-813f-a9e9d6d9e67a\") " pod="e2e-models-e2e-worker-w5-19bf5b/premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b" Nov 04 04:59:13.202790 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:13.202773 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-home\") pod \"premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b\" (UID: \"ac06b52c-cb97-479d-813f-a9e9d6d9e67a\") " pod="e2e-models-e2e-worker-w5-19bf5b/premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b" Nov 04 04:59:13.203076 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:13.203045 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-tmp-dir\") pod \"premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b\" (UID: \"ac06b52c-cb97-479d-813f-a9e9d6d9e67a\") " pod="e2e-models-e2e-worker-w5-19bf5b/premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b" Nov 04 04:59:13.203076 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:13.203056 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-model-cache\") pod \"premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b\" (UID: \"ac06b52c-cb97-479d-813f-a9e9d6d9e67a\") " pod="e2e-models-e2e-worker-w5-19bf5b/premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b" Nov 04 04:59:13.203182 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:13.203167 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-home\") pod \"premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b\" (UID: \"ac06b52c-cb97-479d-813f-a9e9d6d9e67a\") " pod="e2e-models-e2e-worker-w5-19bf5b/premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b" Nov 04 04:59:13.204955 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:13.204932 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-dshm\") pod \"premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b\" (UID: \"ac06b52c-cb97-479d-813f-a9e9d6d9e67a\") " pod="e2e-models-e2e-worker-w5-19bf5b/premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b" Nov 04 04:59:13.205121 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:13.205104 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-tls-certs\") pod \"premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b\" (UID: \"ac06b52c-cb97-479d-813f-a9e9d6d9e67a\") " pod="e2e-models-e2e-worker-w5-19bf5b/premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b" Nov 04 04:59:13.263136 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:13.263077 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrq64\" (UniqueName: \"kubernetes.io/projected/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-kube-api-access-zrq64\") pod \"premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b\" (UID: \"ac06b52c-cb97-479d-813f-a9e9d6d9e67a\") " pod="e2e-models-e2e-worker-w5-19bf5b/premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b" Nov 04 04:59:13.313100 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:13.313070 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-19bf5b/premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b" Nov 04 04:59:13.692655 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:13.692626 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-19bf5b/premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b"] Nov 04 04:59:13.694074 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:59:13.694046 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac06b52c_cb97_479d_813f_a9e9d6d9e67a.slice/crio-5f2e269cbf7447b81b48a184f3a079f74199c8de2a34b0a6a79bc7e4ec48006c WatchSource:0}: Error finding container 5f2e269cbf7447b81b48a184f3a079f74199c8de2a34b0a6a79bc7e4ec48006c: Status 404 returned error can't find the container with id 5f2e269cbf7447b81b48a184f3a079f74199c8de2a34b0a6a79bc7e4ec48006c Nov 04 04:59:13.796673 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:13.796643 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-19bf5b/premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b" event={"ID":"ac06b52c-cb97-479d-813f-a9e9d6d9e67a","Type":"ContainerStarted","Data":"4ec74aec84bce2314467d02a0e4dd56849cc9f7bba750627c753a0e812b8caf6"} Nov 04 04:59:13.796829 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:13.796685 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-19bf5b/premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b" event={"ID":"ac06b52c-cb97-479d-813f-a9e9d6d9e67a","Type":"ContainerStarted","Data":"5f2e269cbf7447b81b48a184f3a079f74199c8de2a34b0a6a79bc7e4ec48006c"} Nov 04 04:59:13.908602 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:13.908552 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-19bf5b/premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b" podStartSLOduration=1.908538526 podStartE2EDuration="1.908538526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:59:13.9058922 +0000 UTC m=+795.595974648" watchObservedRunningTime="2026-11-04 04:59:13.908538526 +0000 UTC m=+795.598620973" Nov 04 04:59:14.313305 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:14.313277 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-19bf5b/premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b" Nov 04 04:59:23.313234 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:23.313197 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-19bf5b/premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b" Nov 04 04:59:23.326251 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:23.326227 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-19bf5b/premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b" Nov 04 04:59:23.877874 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:23.877838 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-19bf5b/premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b" Nov 04 04:59:28.386491 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:28.386455 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-204fa0/premium-w0-204fa0-kserve-6848b499d9-cczvl"] Nov 04 04:59:28.390130 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:28.390110 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-204fa0/premium-w0-204fa0-kserve-6848b499d9-cczvl" Nov 04 04:59:28.485189 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:28.485164 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-204fa0\"/\"premium-w0-204fa0-kserve-self-signed-certs\"" Nov 04 04:59:28.487990 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:28.487964 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-204fa0/premium-w0-204fa0-kserve-6848b499d9-cczvl"] Nov 04 04:59:28.557315 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:28.557270 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37ed0b35-a2e9-49eb-95c5-793da9836c6f-home\") pod \"premium-w0-204fa0-kserve-6848b499d9-cczvl\" (UID: \"37ed0b35-a2e9-49eb-95c5-793da9836c6f\") " pod="e2e-models-e2e-worker-w0-204fa0/premium-w0-204fa0-kserve-6848b499d9-cczvl" Nov 04 04:59:28.557495 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:28.557454 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngcg5\" (UniqueName: \"kubernetes.io/projected/37ed0b35-a2e9-49eb-95c5-793da9836c6f-kube-api-access-ngcg5\") pod \"premium-w0-204fa0-kserve-6848b499d9-cczvl\" (UID: \"37ed0b35-a2e9-49eb-95c5-793da9836c6f\") " pod="e2e-models-e2e-worker-w0-204fa0/premium-w0-204fa0-kserve-6848b499d9-cczvl" Nov 04 04:59:28.557710 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:28.557687 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37ed0b35-a2e9-49eb-95c5-793da9836c6f-tls-certs\") pod \"premium-w0-204fa0-kserve-6848b499d9-cczvl\" (UID: \"37ed0b35-a2e9-49eb-95c5-793da9836c6f\") " pod="e2e-models-e2e-worker-w0-204fa0/premium-w0-204fa0-kserve-6848b499d9-cczvl" Nov 04 04:59:28.557814 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:28.557780 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37ed0b35-a2e9-49eb-95c5-793da9836c6f-tmp-dir\") pod \"premium-w0-204fa0-kserve-6848b499d9-cczvl\" (UID: \"37ed0b35-a2e9-49eb-95c5-793da9836c6f\") " pod="e2e-models-e2e-worker-w0-204fa0/premium-w0-204fa0-kserve-6848b499d9-cczvl" Nov 04 04:59:28.558040 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:28.558017 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37ed0b35-a2e9-49eb-95c5-793da9836c6f-dshm\") pod \"premium-w0-204fa0-kserve-6848b499d9-cczvl\" (UID: \"37ed0b35-a2e9-49eb-95c5-793da9836c6f\") " pod="e2e-models-e2e-worker-w0-204fa0/premium-w0-204fa0-kserve-6848b499d9-cczvl" Nov 04 04:59:28.558214 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:28.558193 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37ed0b35-a2e9-49eb-95c5-793da9836c6f-model-cache\") pod \"premium-w0-204fa0-kserve-6848b499d9-cczvl\" (UID: \"37ed0b35-a2e9-49eb-95c5-793da9836c6f\") " pod="e2e-models-e2e-worker-w0-204fa0/premium-w0-204fa0-kserve-6848b499d9-cczvl" Nov 04 04:59:28.659504 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:28.659428 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37ed0b35-a2e9-49eb-95c5-793da9836c6f-tls-certs\") pod \"premium-w0-204fa0-kserve-6848b499d9-cczvl\" (UID: \"37ed0b35-a2e9-49eb-95c5-793da9836c6f\") " pod="e2e-models-e2e-worker-w0-204fa0/premium-w0-204fa0-kserve-6848b499d9-cczvl" Nov 04 04:59:28.659504 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:28.659485 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37ed0b35-a2e9-49eb-95c5-793da9836c6f-tmp-dir\") pod \"premium-w0-204fa0-kserve-6848b499d9-cczvl\" (UID: \"37ed0b35-a2e9-49eb-95c5-793da9836c6f\") " pod="e2e-models-e2e-worker-w0-204fa0/premium-w0-204fa0-kserve-6848b499d9-cczvl" Nov 04 04:59:28.659716 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:28.659522 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37ed0b35-a2e9-49eb-95c5-793da9836c6f-dshm\") pod \"premium-w0-204fa0-kserve-6848b499d9-cczvl\" (UID: \"37ed0b35-a2e9-49eb-95c5-793da9836c6f\") " pod="e2e-models-e2e-worker-w0-204fa0/premium-w0-204fa0-kserve-6848b499d9-cczvl" Nov 04 04:59:28.659716 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:28.659571 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37ed0b35-a2e9-49eb-95c5-793da9836c6f-model-cache\") pod \"premium-w0-204fa0-kserve-6848b499d9-cczvl\" (UID: \"37ed0b35-a2e9-49eb-95c5-793da9836c6f\") " pod="e2e-models-e2e-worker-w0-204fa0/premium-w0-204fa0-kserve-6848b499d9-cczvl" Nov 04 04:59:28.659716 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:28.659631 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37ed0b35-a2e9-49eb-95c5-793da9836c6f-home\") pod \"premium-w0-204fa0-kserve-6848b499d9-cczvl\" (UID: \"37ed0b35-a2e9-49eb-95c5-793da9836c6f\") " pod="e2e-models-e2e-worker-w0-204fa0/premium-w0-204fa0-kserve-6848b499d9-cczvl" Nov 04 04:59:28.659716 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:28.659710 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngcg5\" (UniqueName: \"kubernetes.io/projected/37ed0b35-a2e9-49eb-95c5-793da9836c6f-kube-api-access-ngcg5\") pod \"premium-w0-204fa0-kserve-6848b499d9-cczvl\" (UID: \"37ed0b35-a2e9-49eb-95c5-793da9836c6f\") " pod="e2e-models-e2e-worker-w0-204fa0/premium-w0-204fa0-kserve-6848b499d9-cczvl" Nov 04 04:59:28.660041 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:28.660018 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37ed0b35-a2e9-49eb-95c5-793da9836c6f-model-cache\") pod \"premium-w0-204fa0-kserve-6848b499d9-cczvl\" (UID: \"37ed0b35-a2e9-49eb-95c5-793da9836c6f\") " pod="e2e-models-e2e-worker-w0-204fa0/premium-w0-204fa0-kserve-6848b499d9-cczvl" Nov 04 04:59:28.660107 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:28.660028 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37ed0b35-a2e9-49eb-95c5-793da9836c6f-tmp-dir\") pod \"premium-w0-204fa0-kserve-6848b499d9-cczvl\" (UID: \"37ed0b35-a2e9-49eb-95c5-793da9836c6f\") " pod="e2e-models-e2e-worker-w0-204fa0/premium-w0-204fa0-kserve-6848b499d9-cczvl" Nov 04 04:59:28.660107 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:28.660080 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37ed0b35-a2e9-49eb-95c5-793da9836c6f-home\") pod \"premium-w0-204fa0-kserve-6848b499d9-cczvl\" (UID: \"37ed0b35-a2e9-49eb-95c5-793da9836c6f\") " pod="e2e-models-e2e-worker-w0-204fa0/premium-w0-204fa0-kserve-6848b499d9-cczvl" Nov 04 04:59:28.661746 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:28.661726 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37ed0b35-a2e9-49eb-95c5-793da9836c6f-dshm\") pod \"premium-w0-204fa0-kserve-6848b499d9-cczvl\" (UID: \"37ed0b35-a2e9-49eb-95c5-793da9836c6f\") " pod="e2e-models-e2e-worker-w0-204fa0/premium-w0-204fa0-kserve-6848b499d9-cczvl" Nov 04 04:59:28.662032 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:28.662014 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37ed0b35-a2e9-49eb-95c5-793da9836c6f-tls-certs\") pod \"premium-w0-204fa0-kserve-6848b499d9-cczvl\" (UID: \"37ed0b35-a2e9-49eb-95c5-793da9836c6f\") " pod="e2e-models-e2e-worker-w0-204fa0/premium-w0-204fa0-kserve-6848b499d9-cczvl" Nov 04 04:59:28.717026 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:28.717003 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngcg5\" (UniqueName: \"kubernetes.io/projected/37ed0b35-a2e9-49eb-95c5-793da9836c6f-kube-api-access-ngcg5\") pod \"premium-w0-204fa0-kserve-6848b499d9-cczvl\" (UID: \"37ed0b35-a2e9-49eb-95c5-793da9836c6f\") " pod="e2e-models-e2e-worker-w0-204fa0/premium-w0-204fa0-kserve-6848b499d9-cczvl" Nov 04 04:59:29.001230 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:29.001157 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-204fa0/premium-w0-204fa0-kserve-6848b499d9-cczvl" Nov 04 04:59:29.377689 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:29.377660 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-204fa0/premium-w0-204fa0-kserve-6848b499d9-cczvl"] Nov 04 04:59:29.378738 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:59:29.378708 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37ed0b35_a2e9_49eb_95c5_793da9836c6f.slice/crio-2b5ae8f393aa5ba69b0e9a87794756499a10aed835f96ce1d5929f803dd8b308 WatchSource:0}: Error finding container 2b5ae8f393aa5ba69b0e9a87794756499a10aed835f96ce1d5929f803dd8b308: Status 404 returned error can't find the container with id 2b5ae8f393aa5ba69b0e9a87794756499a10aed835f96ce1d5929f803dd8b308 Nov 04 04:59:29.901727 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:29.901692 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-204fa0/premium-w0-204fa0-kserve-6848b499d9-cczvl" event={"ID":"37ed0b35-a2e9-49eb-95c5-793da9836c6f","Type":"ContainerStarted","Data":"85452ff402f1088b830a0533632656779de6e5e054f427f4ca13b53e43e624fc"} Nov 04 04:59:29.902106 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:29.901737 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-204fa0/premium-w0-204fa0-kserve-6848b499d9-cczvl" event={"ID":"37ed0b35-a2e9-49eb-95c5-793da9836c6f","Type":"ContainerStarted","Data":"2b5ae8f393aa5ba69b0e9a87794756499a10aed835f96ce1d5929f803dd8b308"} Nov 04 04:59:30.001622 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:30.001599 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-204fa0/premium-w0-204fa0-kserve-6848b499d9-cczvl" Nov 04 04:59:30.013705 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:30.013655 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-204fa0/premium-w0-204fa0-kserve-6848b499d9-cczvl" podStartSLOduration=2.013641976 podStartE2EDuration="2.013641976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:59:30.011152419 +0000 UTC m=+811.701234868" watchObservedRunningTime="2026-11-04 04:59:30.013641976 +0000 UTC m=+811.703724422" Nov 04 04:59:33.456861 ip-10-0-134-224 kubenswrapper[4395]: E1104 04:59:33.456813 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 04:59:39.002339 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:39.002305 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-204fa0/premium-w0-204fa0-kserve-6848b499d9-cczvl" Nov 04 04:59:39.014846 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:39.014825 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-204fa0/premium-w0-204fa0-kserve-6848b499d9-cczvl" Nov 04 04:59:39.969730 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:39.969697 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-204fa0/premium-w0-204fa0-kserve-6848b499d9-cczvl" Nov 04 04:59:40.698338 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:40.698287 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-db8fae/premium-w6-db8fae-kserve-6c464dbc8b-xvnsz"] Nov 04 04:59:40.702501 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:40.702474 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-db8fae/premium-w6-db8fae-kserve-6c464dbc8b-xvnsz" Nov 04 04:59:40.780537 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:40.780500 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm846\" (UniqueName: \"kubernetes.io/projected/002e54f5-f516-44b1-8145-a26084b77264-kube-api-access-jm846\") pod \"premium-w6-db8fae-kserve-6c464dbc8b-xvnsz\" (UID: \"002e54f5-f516-44b1-8145-a26084b77264\") " pod="e2e-models-e2e-worker-w6-db8fae/premium-w6-db8fae-kserve-6c464dbc8b-xvnsz" Nov 04 04:59:40.780698 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:40.780634 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/002e54f5-f516-44b1-8145-a26084b77264-home\") pod \"premium-w6-db8fae-kserve-6c464dbc8b-xvnsz\" (UID: \"002e54f5-f516-44b1-8145-a26084b77264\") " pod="e2e-models-e2e-worker-w6-db8fae/premium-w6-db8fae-kserve-6c464dbc8b-xvnsz" Nov 04 04:59:40.780698 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:40.780662 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/002e54f5-f516-44b1-8145-a26084b77264-model-cache\") pod \"premium-w6-db8fae-kserve-6c464dbc8b-xvnsz\" (UID: \"002e54f5-f516-44b1-8145-a26084b77264\") " pod="e2e-models-e2e-worker-w6-db8fae/premium-w6-db8fae-kserve-6c464dbc8b-xvnsz" Nov 04 04:59:40.780827 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:40.780720 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/002e54f5-f516-44b1-8145-a26084b77264-dshm\") pod \"premium-w6-db8fae-kserve-6c464dbc8b-xvnsz\" (UID: \"002e54f5-f516-44b1-8145-a26084b77264\") " pod="e2e-models-e2e-worker-w6-db8fae/premium-w6-db8fae-kserve-6c464dbc8b-xvnsz" Nov 04 04:59:40.780827 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:40.780759 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/002e54f5-f516-44b1-8145-a26084b77264-tmp-dir\") pod \"premium-w6-db8fae-kserve-6c464dbc8b-xvnsz\" (UID: \"002e54f5-f516-44b1-8145-a26084b77264\") " pod="e2e-models-e2e-worker-w6-db8fae/premium-w6-db8fae-kserve-6c464dbc8b-xvnsz" Nov 04 04:59:40.780827 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:40.780786 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/002e54f5-f516-44b1-8145-a26084b77264-tls-certs\") pod \"premium-w6-db8fae-kserve-6c464dbc8b-xvnsz\" (UID: \"002e54f5-f516-44b1-8145-a26084b77264\") " pod="e2e-models-e2e-worker-w6-db8fae/premium-w6-db8fae-kserve-6c464dbc8b-xvnsz" Nov 04 04:59:40.797320 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:40.797270 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-db8fae\"/\"kube-root-ca.crt\"" Nov 04 04:59:40.797460 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:40.797392 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-db8fae\"/\"default-dockercfg-zjlmn\"" Nov 04 04:59:40.797460 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:40.797398 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-db8fae\"/\"openshift-service-ca.crt\"" Nov 04 04:59:40.797460 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:40.797408 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-db8fae\"/\"premium-w6-db8fae-kserve-self-signed-certs\"" Nov 04 04:59:40.800036 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:40.800013 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-db8fae/premium-w6-db8fae-kserve-6c464dbc8b-xvnsz"] Nov 04 04:59:40.881625 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:40.881596 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm846\" (UniqueName: \"kubernetes.io/projected/002e54f5-f516-44b1-8145-a26084b77264-kube-api-access-jm846\") pod \"premium-w6-db8fae-kserve-6c464dbc8b-xvnsz\" (UID: \"002e54f5-f516-44b1-8145-a26084b77264\") " pod="e2e-models-e2e-worker-w6-db8fae/premium-w6-db8fae-kserve-6c464dbc8b-xvnsz" Nov 04 04:59:40.881792 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:40.881673 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/002e54f5-f516-44b1-8145-a26084b77264-home\") pod \"premium-w6-db8fae-kserve-6c464dbc8b-xvnsz\" (UID: \"002e54f5-f516-44b1-8145-a26084b77264\") " pod="e2e-models-e2e-worker-w6-db8fae/premium-w6-db8fae-kserve-6c464dbc8b-xvnsz" Nov 04 04:59:40.881792 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:40.881692 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/002e54f5-f516-44b1-8145-a26084b77264-model-cache\") pod \"premium-w6-db8fae-kserve-6c464dbc8b-xvnsz\" (UID: \"002e54f5-f516-44b1-8145-a26084b77264\") " pod="e2e-models-e2e-worker-w6-db8fae/premium-w6-db8fae-kserve-6c464dbc8b-xvnsz" Nov 04 04:59:40.881792 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:40.881711 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/002e54f5-f516-44b1-8145-a26084b77264-dshm\") pod \"premium-w6-db8fae-kserve-6c464dbc8b-xvnsz\" (UID: \"002e54f5-f516-44b1-8145-a26084b77264\") " pod="e2e-models-e2e-worker-w6-db8fae/premium-w6-db8fae-kserve-6c464dbc8b-xvnsz" Nov 04 04:59:40.881792 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:40.881725 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/002e54f5-f516-44b1-8145-a26084b77264-tmp-dir\") pod \"premium-w6-db8fae-kserve-6c464dbc8b-xvnsz\" (UID: \"002e54f5-f516-44b1-8145-a26084b77264\") " pod="e2e-models-e2e-worker-w6-db8fae/premium-w6-db8fae-kserve-6c464dbc8b-xvnsz" Nov 04 04:59:40.881792 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:40.881745 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/002e54f5-f516-44b1-8145-a26084b77264-tls-certs\") pod \"premium-w6-db8fae-kserve-6c464dbc8b-xvnsz\" (UID: \"002e54f5-f516-44b1-8145-a26084b77264\") " pod="e2e-models-e2e-worker-w6-db8fae/premium-w6-db8fae-kserve-6c464dbc8b-xvnsz" Nov 04 04:59:40.882101 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:40.882081 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/002e54f5-f516-44b1-8145-a26084b77264-model-cache\") pod \"premium-w6-db8fae-kserve-6c464dbc8b-xvnsz\" (UID: \"002e54f5-f516-44b1-8145-a26084b77264\") " pod="e2e-models-e2e-worker-w6-db8fae/premium-w6-db8fae-kserve-6c464dbc8b-xvnsz" Nov 04 04:59:40.882213 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:40.882187 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/002e54f5-f516-44b1-8145-a26084b77264-home\") pod \"premium-w6-db8fae-kserve-6c464dbc8b-xvnsz\" (UID: \"002e54f5-f516-44b1-8145-a26084b77264\") " pod="e2e-models-e2e-worker-w6-db8fae/premium-w6-db8fae-kserve-6c464dbc8b-xvnsz" Nov 04 04:59:40.882391 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:40.882277 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/002e54f5-f516-44b1-8145-a26084b77264-tmp-dir\") pod \"premium-w6-db8fae-kserve-6c464dbc8b-xvnsz\" (UID: \"002e54f5-f516-44b1-8145-a26084b77264\") " pod="e2e-models-e2e-worker-w6-db8fae/premium-w6-db8fae-kserve-6c464dbc8b-xvnsz" Nov 04 04:59:40.884060 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:40.884038 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/002e54f5-f516-44b1-8145-a26084b77264-dshm\") pod \"premium-w6-db8fae-kserve-6c464dbc8b-xvnsz\" (UID: \"002e54f5-f516-44b1-8145-a26084b77264\") " pod="e2e-models-e2e-worker-w6-db8fae/premium-w6-db8fae-kserve-6c464dbc8b-xvnsz" Nov 04 04:59:40.884502 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:40.884482 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/002e54f5-f516-44b1-8145-a26084b77264-tls-certs\") pod \"premium-w6-db8fae-kserve-6c464dbc8b-xvnsz\" (UID: \"002e54f5-f516-44b1-8145-a26084b77264\") " pod="e2e-models-e2e-worker-w6-db8fae/premium-w6-db8fae-kserve-6c464dbc8b-xvnsz" Nov 04 04:59:40.939110 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:40.939075 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm846\" (UniqueName: \"kubernetes.io/projected/002e54f5-f516-44b1-8145-a26084b77264-kube-api-access-jm846\") pod \"premium-w6-db8fae-kserve-6c464dbc8b-xvnsz\" (UID: \"002e54f5-f516-44b1-8145-a26084b77264\") " pod="e2e-models-e2e-worker-w6-db8fae/premium-w6-db8fae-kserve-6c464dbc8b-xvnsz" Nov 04 04:59:41.013545 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:41.013483 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-db8fae/premium-w6-db8fae-kserve-6c464dbc8b-xvnsz" Nov 04 04:59:41.396515 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:41.396488 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-db8fae/premium-w6-db8fae-kserve-6c464dbc8b-xvnsz"] Nov 04 04:59:41.398042 ip-10-0-134-224 kubenswrapper[4395]: W1104 04:59:41.398014 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod002e54f5_f516_44b1_8145_a26084b77264.slice/crio-5fc52f0ef4d5e73fd100fd44aaffc1abec85794a4f2aaba9b9ed69db996f853e WatchSource:0}: Error finding container 5fc52f0ef4d5e73fd100fd44aaffc1abec85794a4f2aaba9b9ed69db996f853e: Status 404 returned error can't find the container with id 5fc52f0ef4d5e73fd100fd44aaffc1abec85794a4f2aaba9b9ed69db996f853e Nov 04 04:59:41.979892 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:41.979859 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-db8fae/premium-w6-db8fae-kserve-6c464dbc8b-xvnsz" event={"ID":"002e54f5-f516-44b1-8145-a26084b77264","Type":"ContainerStarted","Data":"c08e3ddc635f2d165c9975db699524b2ffa69df63a32d4d8234a6104ed91f3f7"} Nov 04 04:59:41.980284 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:41.979900 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-db8fae/premium-w6-db8fae-kserve-6c464dbc8b-xvnsz" event={"ID":"002e54f5-f516-44b1-8145-a26084b77264","Type":"ContainerStarted","Data":"5fc52f0ef4d5e73fd100fd44aaffc1abec85794a4f2aaba9b9ed69db996f853e"} Nov 04 04:59:42.014533 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:42.014506 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-db8fae/premium-w6-db8fae-kserve-6c464dbc8b-xvnsz" Nov 04 04:59:42.095795 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:42.095753 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-db8fae/premium-w6-db8fae-kserve-6c464dbc8b-xvnsz" podStartSLOduration=2.095741124 podStartE2EDuration="2.095741124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 04:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 04:59:42.094510088 +0000 UTC m=+823.784592535" watchObservedRunningTime="2026-11-04 04:59:42.095741124 +0000 UTC m=+823.785823571" Nov 04 04:59:51.013625 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:51.013594 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-db8fae/premium-w6-db8fae-kserve-6c464dbc8b-xvnsz" Nov 04 04:59:51.025794 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:51.025769 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-db8fae/premium-w6-db8fae-kserve-6c464dbc8b-xvnsz" Nov 04 04:59:51.046734 ip-10-0-134-224 kubenswrapper[4395]: I1104 04:59:51.046711 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-db8fae/premium-w6-db8fae-kserve-6c464dbc8b-xvnsz" Nov 04 05:00:00.170446 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.170413 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-db8fae-29896140-sp67z"] Nov 04 05:00:00.174379 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.174359 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c5bb6d-29896140-nk784"] Nov 04 05:00:00.174503 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.174477 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-db8fae-29896140-sp67z" Nov 04 05:00:00.177830 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.177811 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c5bb6d-29896140-nk784" Nov 04 05:00:00.191383 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.191362 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-204fa0-29896140-7xcmh"] Nov 04 05:00:00.195525 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.195507 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-204fa0-29896140-7xcmh" Nov 04 05:00:00.216562 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.216539 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-204e38-29896140-vd5n7"] Nov 04 05:00:00.220026 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.220012 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-204e38-29896140-vd5n7" Nov 04 05:00:00.223437 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.223419 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nz4vh\"" Nov 04 05:00:00.274657 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.274636 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c5bb6d-29896140-nk784"] Nov 04 05:00:00.277609 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.277577 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-db8fae-29896140-sp67z"] Nov 04 05:00:00.297746 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.297718 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb6bcee-a132-4152-aea6-7164e9a6e003-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-db8fae-29896140-sp67z\" (UID: \"eeb6bcee-a132-4152-aea6-7164e9a6e003\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-db8fae-29896140-sp67z" Nov 04 05:00:00.297858 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.297760 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr9ll\" (UniqueName: \"kubernetes.io/projected/aa369b32-590d-4433-938d-987b043ae4a2-kube-api-access-cr9ll\") pod \"maas-api-key-cleanup-e2e-worker-w0-204fa0-29896140-7xcmh\" (UID: \"aa369b32-590d-4433-938d-987b043ae4a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-204fa0-29896140-7xcmh" Nov 04 05:00:00.297858 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.297789 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa369b32-590d-4433-938d-987b043ae4a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-204fa0-29896140-7xcmh\" (UID: \"aa369b32-590d-4433-938d-987b043ae4a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-204fa0-29896140-7xcmh" Nov 04 05:00:00.297981 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.297933 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/211196ed-10bb-4595-a811-0151d6b13889-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c5bb6d-29896140-nk784\" (UID: \"211196ed-10bb-4595-a811-0151d6b13889\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c5bb6d-29896140-nk784" Nov 04 05:00:00.298059 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.297995 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7pwf\" (UniqueName: \"kubernetes.io/projected/eeb6bcee-a132-4152-aea6-7164e9a6e003-kube-api-access-j7pwf\") pod \"maas-api-key-cleanup-e2e-worker-w6-db8fae-29896140-sp67z\" (UID: \"eeb6bcee-a132-4152-aea6-7164e9a6e003\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-db8fae-29896140-sp67z" Nov 04 05:00:00.298059 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.298042 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdsxs\" (UniqueName: \"kubernetes.io/projected/211196ed-10bb-4595-a811-0151d6b13889-kube-api-access-xdsxs\") pod \"maas-api-key-cleanup-e2e-worker-w2-c5bb6d-29896140-nk784\" (UID: \"211196ed-10bb-4595-a811-0151d6b13889\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c5bb6d-29896140-nk784" Nov 04 05:00:00.322379 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.322356 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-204fa0-29896140-7xcmh"] Nov 04 05:00:00.325488 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.325464 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-204e38-29896140-vd5n7"] Nov 04 05:00:00.399278 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.399251 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa369b32-590d-4433-938d-987b043ae4a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-204fa0-29896140-7xcmh\" (UID: \"aa369b32-590d-4433-938d-987b043ae4a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-204fa0-29896140-7xcmh" Nov 04 05:00:00.399278 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.399281 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv625\" (UniqueName: \"kubernetes.io/projected/ae07bf50-a836-4192-85a4-ebee4dd2a3af-kube-api-access-sv625\") pod \"maas-api-key-cleanup-e2e-worker-w3-204e38-29896140-vd5n7\" (UID: \"ae07bf50-a836-4192-85a4-ebee4dd2a3af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-204e38-29896140-vd5n7" Nov 04 05:00:00.399470 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.399360 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/211196ed-10bb-4595-a811-0151d6b13889-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c5bb6d-29896140-nk784\" (UID: \"211196ed-10bb-4595-a811-0151d6b13889\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c5bb6d-29896140-nk784" Nov 04 05:00:00.399913 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.399887 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae07bf50-a836-4192-85a4-ebee4dd2a3af-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-204e38-29896140-vd5n7\" (UID: \"ae07bf50-a836-4192-85a4-ebee4dd2a3af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-204e38-29896140-vd5n7" Nov 04 05:00:00.400020 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.399954 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7pwf\" (UniqueName: \"kubernetes.io/projected/eeb6bcee-a132-4152-aea6-7164e9a6e003-kube-api-access-j7pwf\") pod \"maas-api-key-cleanup-e2e-worker-w6-db8fae-29896140-sp67z\" (UID: \"eeb6bcee-a132-4152-aea6-7164e9a6e003\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-db8fae-29896140-sp67z" Nov 04 05:00:00.400091 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.400013 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdsxs\" (UniqueName: \"kubernetes.io/projected/211196ed-10bb-4595-a811-0151d6b13889-kube-api-access-xdsxs\") pod \"maas-api-key-cleanup-e2e-worker-w2-c5bb6d-29896140-nk784\" (UID: \"211196ed-10bb-4595-a811-0151d6b13889\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c5bb6d-29896140-nk784" Nov 04 05:00:00.400091 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.400044 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa369b32-590d-4433-938d-987b043ae4a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-204fa0-29896140-7xcmh\" (UID: \"aa369b32-590d-4433-938d-987b043ae4a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-204fa0-29896140-7xcmh" Nov 04 05:00:00.400215 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.400133 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb6bcee-a132-4152-aea6-7164e9a6e003-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-db8fae-29896140-sp67z\" (UID: \"eeb6bcee-a132-4152-aea6-7164e9a6e003\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-db8fae-29896140-sp67z" Nov 04 05:00:00.400215 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.400180 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/211196ed-10bb-4595-a811-0151d6b13889-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c5bb6d-29896140-nk784\" (UID: \"211196ed-10bb-4595-a811-0151d6b13889\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c5bb6d-29896140-nk784" Nov 04 05:00:00.400215 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.400193 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr9ll\" (UniqueName: \"kubernetes.io/projected/aa369b32-590d-4433-938d-987b043ae4a2-kube-api-access-cr9ll\") pod \"maas-api-key-cleanup-e2e-worker-w0-204fa0-29896140-7xcmh\" (UID: \"aa369b32-590d-4433-938d-987b043ae4a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-204fa0-29896140-7xcmh" Nov 04 05:00:00.400837 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.400813 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb6bcee-a132-4152-aea6-7164e9a6e003-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-db8fae-29896140-sp67z\" (UID: \"eeb6bcee-a132-4152-aea6-7164e9a6e003\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-db8fae-29896140-sp67z" Nov 04 05:00:00.456991 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.456929 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7pwf\" (UniqueName: \"kubernetes.io/projected/eeb6bcee-a132-4152-aea6-7164e9a6e003-kube-api-access-j7pwf\") pod \"maas-api-key-cleanup-e2e-worker-w6-db8fae-29896140-sp67z\" (UID: \"eeb6bcee-a132-4152-aea6-7164e9a6e003\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-db8fae-29896140-sp67z" Nov 04 05:00:00.457921 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.457898 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdsxs\" (UniqueName: \"kubernetes.io/projected/211196ed-10bb-4595-a811-0151d6b13889-kube-api-access-xdsxs\") pod \"maas-api-key-cleanup-e2e-worker-w2-c5bb6d-29896140-nk784\" (UID: \"211196ed-10bb-4595-a811-0151d6b13889\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c5bb6d-29896140-nk784" Nov 04 05:00:00.458047 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.458028 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr9ll\" (UniqueName: \"kubernetes.io/projected/aa369b32-590d-4433-938d-987b043ae4a2-kube-api-access-cr9ll\") pod \"maas-api-key-cleanup-e2e-worker-w0-204fa0-29896140-7xcmh\" (UID: \"aa369b32-590d-4433-938d-987b043ae4a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-204fa0-29896140-7xcmh" Nov 04 05:00:00.488952 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.488929 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-db8fae-29896140-sp67z" Nov 04 05:00:00.494811 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.494790 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c5bb6d-29896140-nk784" Nov 04 05:00:00.501374 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.501347 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv625\" (UniqueName: \"kubernetes.io/projected/ae07bf50-a836-4192-85a4-ebee4dd2a3af-kube-api-access-sv625\") pod \"maas-api-key-cleanup-e2e-worker-w3-204e38-29896140-vd5n7\" (UID: \"ae07bf50-a836-4192-85a4-ebee4dd2a3af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-204e38-29896140-vd5n7" Nov 04 05:00:00.501497 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.501479 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae07bf50-a836-4192-85a4-ebee4dd2a3af-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-204e38-29896140-vd5n7\" (UID: \"ae07bf50-a836-4192-85a4-ebee4dd2a3af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-204e38-29896140-vd5n7" Nov 04 05:00:00.502292 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.502263 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae07bf50-a836-4192-85a4-ebee4dd2a3af-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-204e38-29896140-vd5n7\" (UID: \"ae07bf50-a836-4192-85a4-ebee4dd2a3af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-204e38-29896140-vd5n7" Nov 04 05:00:00.508216 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.508183 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-204fa0-29896140-7xcmh" Nov 04 05:00:00.566496 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.563041 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv625\" (UniqueName: \"kubernetes.io/projected/ae07bf50-a836-4192-85a4-ebee4dd2a3af-kube-api-access-sv625\") pod \"maas-api-key-cleanup-e2e-worker-w3-204e38-29896140-vd5n7\" (UID: \"ae07bf50-a836-4192-85a4-ebee4dd2a3af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-204e38-29896140-vd5n7" Nov 04 05:00:00.832200 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.832161 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-204e38-29896140-vd5n7" Nov 04 05:00:00.873552 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.873523 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-db8fae-29896140-sp67z"] Nov 04 05:00:00.928241 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.928216 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c5bb6d-29896140-nk784"] Nov 04 05:00:00.930969 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:00.930944 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-204fa0-29896140-7xcmh"] Nov 04 05:00:00.931800 ip-10-0-134-224 kubenswrapper[4395]: W1104 05:00:00.931775 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa369b32_590d_4433_938d_987b043ae4a2.slice/crio-5a9efbb0a26fa64335ea361a8dc733d739decf77fd0d54f3785eed4b740ba1bb WatchSource:0}: Error finding container 5a9efbb0a26fa64335ea361a8dc733d739decf77fd0d54f3785eed4b740ba1bb: Status 404 returned error can't find the container with id 5a9efbb0a26fa64335ea361a8dc733d739decf77fd0d54f3785eed4b740ba1bb Nov 04 05:00:00.933146 ip-10-0-134-224 kubenswrapper[4395]: W1104 05:00:00.933119 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod211196ed_10bb_4595_a811_0151d6b13889.slice/crio-84b8aa6b297aea22739378d4987f74ae512606d67d594af53709fac9fd0b6c76 WatchSource:0}: Error finding container 84b8aa6b297aea22739378d4987f74ae512606d67d594af53709fac9fd0b6c76: Status 404 returned error can't find the container with id 84b8aa6b297aea22739378d4987f74ae512606d67d594af53709fac9fd0b6c76 Nov 04 05:00:01.121392 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:01.121325 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-db8fae-29896140-sp67z" event={"ID":"eeb6bcee-a132-4152-aea6-7164e9a6e003","Type":"ContainerStarted","Data":"099a3e13fe41adc6ae21294ffc4aa61bda4b7ab9cbdd27eb82b8545b01fcde4b"} Nov 04 05:00:01.122605 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:01.122583 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c5bb6d-29896140-nk784" event={"ID":"211196ed-10bb-4595-a811-0151d6b13889","Type":"ContainerStarted","Data":"84b8aa6b297aea22739378d4987f74ae512606d67d594af53709fac9fd0b6c76"} Nov 04 05:00:01.123772 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:01.123746 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-204fa0-29896140-7xcmh" event={"ID":"aa369b32-590d-4433-938d-987b043ae4a2","Type":"ContainerStarted","Data":"5a9efbb0a26fa64335ea361a8dc733d739decf77fd0d54f3785eed4b740ba1bb"} Nov 04 05:00:01.216770 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:01.216747 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-204e38-29896140-vd5n7"] Nov 04 05:00:01.218046 ip-10-0-134-224 kubenswrapper[4395]: W1104 05:00:01.218020 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae07bf50_a836_4192_85a4_ebee4dd2a3af.slice/crio-574306ca8f50e40278ea37cbfbe1bf39b69d49bfb67c37ed20a8a10c3dca7447 WatchSource:0}: Error finding container 574306ca8f50e40278ea37cbfbe1bf39b69d49bfb67c37ed20a8a10c3dca7447: Status 404 returned error can't find the container with id 574306ca8f50e40278ea37cbfbe1bf39b69d49bfb67c37ed20a8a10c3dca7447 Nov 04 05:00:02.132489 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:02.132454 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-204e38-29896140-vd5n7" event={"ID":"ae07bf50-a836-4192-85a4-ebee4dd2a3af","Type":"ContainerStarted","Data":"574306ca8f50e40278ea37cbfbe1bf39b69d49bfb67c37ed20a8a10c3dca7447"} Nov 04 05:00:03.458888 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:00:03.458847 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 05:00:09.185126 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:09.185089 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-204e38-29896140-vd5n7" event={"ID":"ae07bf50-a836-4192-85a4-ebee4dd2a3af","Type":"ContainerStarted","Data":"9b7cef38c9be244abd0521319d2d7d1c0789261f5b8fb1377aeab1c07bab5e6b"} Nov 04 05:00:09.186915 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:09.186888 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c5bb6d-29896140-nk784" event={"ID":"211196ed-10bb-4595-a811-0151d6b13889","Type":"ContainerStarted","Data":"72ced2728df9f86c7fc225b3817c9eca1c13d74f3b888e5e1d5711f99eb19f26"} Nov 04 05:00:09.189189 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:09.188743 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-db8fae-29896140-sp67z" event={"ID":"eeb6bcee-a132-4152-aea6-7164e9a6e003","Type":"ContainerStarted","Data":"585d7468b017942999e986086b0c750f6870c24ebd5cbbdc370a2573e129bdcf"} Nov 04 05:00:09.294128 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:09.294084 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-204e38-29896140-vd5n7" podStartSLOduration=2.18368147 podStartE2EDuration="9.294068792s" totalImagesPullingTime="7.110387322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:00:09.291844871 +0000 UTC m=+850.981927317" watchObservedRunningTime="2026-11-04 05:00:09.294068792 +0000 UTC m=+850.984151241" Nov 04 05:00:09.400241 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:09.400192 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-db8fae-29896140-sp67z" podStartSLOduration=1.7551061749999999 podStartE2EDuration="9.400178924s" totalImagesPullingTime="7.645072749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:00:09.399036657 +0000 UTC m=+851.089119106" watchObservedRunningTime="2026-11-04 05:00:09.400178924 +0000 UTC m=+851.090261370" Nov 04 05:00:09.508861 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:09.508778 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c5bb6d-29896140-nk784" podStartSLOduration=2.2439950140000002 podStartE2EDuration="9.508761482s" totalImagesPullingTime="7.264766468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:00:09.506076102 +0000 UTC m=+851.196158549" watchObservedRunningTime="2026-11-04 05:00:09.508761482 +0000 UTC m=+851.198843930" Nov 04 05:00:10.224765 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:10.224737 4395 scope.go:122] "RemoveContainer" containerID="585d7468b017942999e986086b0c750f6870c24ebd5cbbdc370a2573e129bdcf" Nov 04 05:00:10.225547 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:10.225401 4395 generic.go:356] "Generic (PLEG): container finished" podID="eeb6bcee-a132-4152-aea6-7164e9a6e003" containerID="585d7468b017942999e986086b0c750f6870c24ebd5cbbdc370a2573e129bdcf" exitCode=60 Nov 04 05:00:10.225677 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:10.224287 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-db8fae-29896140-sp67z" event={"ID":"eeb6bcee-a132-4152-aea6-7164e9a6e003","Type":"ContainerDied","Data":"585d7468b017942999e986086b0c750f6870c24ebd5cbbdc370a2573e129bdcf"} Nov 04 05:00:10.229362 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:10.229340 4395 generic.go:356] "Generic (PLEG): container finished" podID="ae07bf50-a836-4192-85a4-ebee4dd2a3af" containerID="9b7cef38c9be244abd0521319d2d7d1c0789261f5b8fb1377aeab1c07bab5e6b" exitCode=60 Nov 04 05:00:10.229548 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:10.229414 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-204e38-29896140-vd5n7" event={"ID":"ae07bf50-a836-4192-85a4-ebee4dd2a3af","Type":"ContainerDied","Data":"9b7cef38c9be244abd0521319d2d7d1c0789261f5b8fb1377aeab1c07bab5e6b"} Nov 04 05:00:10.229698 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:10.229578 4395 scope.go:122] "RemoveContainer" containerID="9b7cef38c9be244abd0521319d2d7d1c0789261f5b8fb1377aeab1c07bab5e6b" Nov 04 05:00:10.234686 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:10.234576 4395 generic.go:356] "Generic (PLEG): container finished" podID="211196ed-10bb-4595-a811-0151d6b13889" containerID="72ced2728df9f86c7fc225b3817c9eca1c13d74f3b888e5e1d5711f99eb19f26" exitCode=60 Nov 04 05:00:10.234686 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:10.234655 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c5bb6d-29896140-nk784" event={"ID":"211196ed-10bb-4595-a811-0151d6b13889","Type":"ContainerDied","Data":"72ced2728df9f86c7fc225b3817c9eca1c13d74f3b888e5e1d5711f99eb19f26"} Nov 04 05:00:10.234898 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:10.234880 4395 scope.go:122] "RemoveContainer" containerID="72ced2728df9f86c7fc225b3817c9eca1c13d74f3b888e5e1d5711f99eb19f26" Nov 04 05:00:10.237570 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:10.237543 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-204fa0-29896140-7xcmh" event={"ID":"aa369b32-590d-4433-938d-987b043ae4a2","Type":"ContainerStarted","Data":"1ec22ddb88878a23b8758178ce8becac1002c41d97de6c632d422f7c3181c8f0"} Nov 04 05:00:10.571585 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:10.571525 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-204fa0-29896140-7xcmh" podStartSLOduration=2.470799917 podStartE2EDuration="10.571504963s" totalImagesPullingTime="8.100705046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:00:10.56365281 +0000 UTC m=+852.253735257" watchObservedRunningTime="2026-11-04 05:00:10.571504963 +0000 UTC m=+852.261587410" Nov 04 05:00:11.245990 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:11.245939 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-db8fae-29896140-sp67z" event={"ID":"eeb6bcee-a132-4152-aea6-7164e9a6e003","Type":"ContainerStarted","Data":"9f26d9f945d84c47138eec55fdf959460744264bf807d05f04eb8a1852216ff3"} Nov 04 05:00:11.248174 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:11.248135 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-204e38-29896140-vd5n7" event={"ID":"ae07bf50-a836-4192-85a4-ebee4dd2a3af","Type":"ContainerStarted","Data":"f614aab3217c852b986e90138950633559089943b7bbeb4e7c719eb2feb31ac7"} Nov 04 05:00:11.250280 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:11.250251 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c5bb6d-29896140-nk784" event={"ID":"211196ed-10bb-4595-a811-0151d6b13889","Type":"ContainerStarted","Data":"70003c3bd44c6a80f2785bc5e6628a2ac462e2abc65819fd634a3ac7940312c1"} Nov 04 05:00:11.251766 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:11.251738 4395 generic.go:356] "Generic (PLEG): container finished" podID="aa369b32-590d-4433-938d-987b043ae4a2" containerID="1ec22ddb88878a23b8758178ce8becac1002c41d97de6c632d422f7c3181c8f0" exitCode=60 Nov 04 05:00:11.251868 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:11.251788 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-204fa0-29896140-7xcmh" event={"ID":"aa369b32-590d-4433-938d-987b043ae4a2","Type":"ContainerDied","Data":"1ec22ddb88878a23b8758178ce8becac1002c41d97de6c632d422f7c3181c8f0"} Nov 04 05:00:11.252039 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:11.252026 4395 scope.go:122] "RemoveContainer" containerID="1ec22ddb88878a23b8758178ce8becac1002c41d97de6c632d422f7c3181c8f0" Nov 04 05:00:12.261912 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:12.261868 4395 generic.go:356] "Generic (PLEG): container finished" podID="ae07bf50-a836-4192-85a4-ebee4dd2a3af" containerID="f614aab3217c852b986e90138950633559089943b7bbeb4e7c719eb2feb31ac7" exitCode=60 Nov 04 05:00:12.262383 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:12.261946 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-204e38-29896140-vd5n7" event={"ID":"ae07bf50-a836-4192-85a4-ebee4dd2a3af","Type":"ContainerDied","Data":"f614aab3217c852b986e90138950633559089943b7bbeb4e7c719eb2feb31ac7"} Nov 04 05:00:12.262383 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:12.262015 4395 scope.go:122] "RemoveContainer" containerID="9b7cef38c9be244abd0521319d2d7d1c0789261f5b8fb1377aeab1c07bab5e6b" Nov 04 05:00:12.262383 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:12.262261 4395 scope.go:122] "RemoveContainer" containerID="f614aab3217c852b986e90138950633559089943b7bbeb4e7c719eb2feb31ac7" Nov 04 05:00:12.262639 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:00:12.262611 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-204e38-29896140-vd5n7_odh-ai-gateway-infra(ae07bf50-a836-4192-85a4-ebee4dd2a3af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-204e38-29896140-vd5n7" podUID="ae07bf50-a836-4192-85a4-ebee4dd2a3af" Nov 04 05:00:12.265247 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:12.265207 4395 generic.go:356] "Generic (PLEG): container finished" podID="211196ed-10bb-4595-a811-0151d6b13889" containerID="70003c3bd44c6a80f2785bc5e6628a2ac462e2abc65819fd634a3ac7940312c1" exitCode=60 Nov 04 05:00:12.265394 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:12.265290 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c5bb6d-29896140-nk784" event={"ID":"211196ed-10bb-4595-a811-0151d6b13889","Type":"ContainerDied","Data":"70003c3bd44c6a80f2785bc5e6628a2ac462e2abc65819fd634a3ac7940312c1"} Nov 04 05:00:12.265532 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:12.265516 4395 scope.go:122] "RemoveContainer" containerID="70003c3bd44c6a80f2785bc5e6628a2ac462e2abc65819fd634a3ac7940312c1" Nov 04 05:00:12.265795 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:00:12.265774 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c5bb6d-29896140-nk784_odh-ai-gateway-infra(211196ed-10bb-4595-a811-0151d6b13889)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c5bb6d-29896140-nk784" podUID="211196ed-10bb-4595-a811-0151d6b13889" Nov 04 05:00:12.267777 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:12.267758 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-204fa0-29896140-7xcmh" event={"ID":"aa369b32-590d-4433-938d-987b043ae4a2","Type":"ContainerStarted","Data":"4a5597ed158f79a7e8878c6770d022a72445e324e50120a2260bf32172187bfd"} Nov 04 05:00:12.270075 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:12.270054 4395 generic.go:356] "Generic (PLEG): container finished" podID="eeb6bcee-a132-4152-aea6-7164e9a6e003" containerID="9f26d9f945d84c47138eec55fdf959460744264bf807d05f04eb8a1852216ff3" exitCode=60 Nov 04 05:00:12.270246 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:12.270138 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-db8fae-29896140-sp67z" event={"ID":"eeb6bcee-a132-4152-aea6-7164e9a6e003","Type":"ContainerDied","Data":"9f26d9f945d84c47138eec55fdf959460744264bf807d05f04eb8a1852216ff3"} Nov 04 05:00:12.270355 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:12.270323 4395 scope.go:122] "RemoveContainer" containerID="9f26d9f945d84c47138eec55fdf959460744264bf807d05f04eb8a1852216ff3" Nov 04 05:00:12.270549 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:00:12.270529 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-db8fae-29896140-sp67z_odh-ai-gateway-infra(eeb6bcee-a132-4152-aea6-7164e9a6e003)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-db8fae-29896140-sp67z" podUID="eeb6bcee-a132-4152-aea6-7164e9a6e003" Nov 04 05:00:12.277578 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:12.277559 4395 scope.go:122] "RemoveContainer" containerID="72ced2728df9f86c7fc225b3817c9eca1c13d74f3b888e5e1d5711f99eb19f26" Nov 04 05:00:12.289887 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:12.289869 4395 scope.go:122] "RemoveContainer" containerID="585d7468b017942999e986086b0c750f6870c24ebd5cbbdc370a2573e129bdcf" Nov 04 05:00:13.297319 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:13.296870 4395 scope.go:122] "RemoveContainer" containerID="9f26d9f945d84c47138eec55fdf959460744264bf807d05f04eb8a1852216ff3" Nov 04 05:00:13.297319 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:00:13.297182 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-db8fae-29896140-sp67z_odh-ai-gateway-infra(eeb6bcee-a132-4152-aea6-7164e9a6e003)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-db8fae-29896140-sp67z" podUID="eeb6bcee-a132-4152-aea6-7164e9a6e003" Nov 04 05:00:13.329142 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:13.329106 4395 scope.go:122] "RemoveContainer" containerID="f614aab3217c852b986e90138950633559089943b7bbeb4e7c719eb2feb31ac7" Nov 04 05:00:13.329465 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:00:13.329443 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-204e38-29896140-vd5n7_odh-ai-gateway-infra(ae07bf50-a836-4192-85a4-ebee4dd2a3af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-204e38-29896140-vd5n7" podUID="ae07bf50-a836-4192-85a4-ebee4dd2a3af" Nov 04 05:00:13.330280 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:13.330248 4395 scope.go:122] "RemoveContainer" containerID="70003c3bd44c6a80f2785bc5e6628a2ac462e2abc65819fd634a3ac7940312c1" Nov 04 05:00:13.330560 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:00:13.330531 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c5bb6d-29896140-nk784_odh-ai-gateway-infra(211196ed-10bb-4595-a811-0151d6b13889)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c5bb6d-29896140-nk784" podUID="211196ed-10bb-4595-a811-0151d6b13889" Nov 04 05:00:13.332376 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:13.332343 4395 generic.go:356] "Generic (PLEG): container finished" podID="aa369b32-590d-4433-938d-987b043ae4a2" containerID="4a5597ed158f79a7e8878c6770d022a72445e324e50120a2260bf32172187bfd" exitCode=60 Nov 04 05:00:13.332492 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:13.332397 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-204fa0-29896140-7xcmh" event={"ID":"aa369b32-590d-4433-938d-987b043ae4a2","Type":"ContainerDied","Data":"4a5597ed158f79a7e8878c6770d022a72445e324e50120a2260bf32172187bfd"} Nov 04 05:00:13.332492 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:13.332447 4395 scope.go:122] "RemoveContainer" containerID="1ec22ddb88878a23b8758178ce8becac1002c41d97de6c632d422f7c3181c8f0" Nov 04 05:00:13.332658 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:13.332636 4395 scope.go:122] "RemoveContainer" containerID="4a5597ed158f79a7e8878c6770d022a72445e324e50120a2260bf32172187bfd" Nov 04 05:00:13.332926 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:00:13.332905 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-204fa0-29896140-7xcmh_odh-ai-gateway-infra(aa369b32-590d-4433-938d-987b043ae4a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-204fa0-29896140-7xcmh" podUID="aa369b32-590d-4433-938d-987b043ae4a2" Nov 04 05:00:14.340447 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:14.340416 4395 scope.go:122] "RemoveContainer" containerID="4a5597ed158f79a7e8878c6770d022a72445e324e50120a2260bf32172187bfd" Nov 04 05:00:14.340801 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:00:14.340662 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-204fa0-29896140-7xcmh_odh-ai-gateway-infra(aa369b32-590d-4433-938d-987b043ae4a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-204fa0-29896140-7xcmh" podUID="aa369b32-590d-4433-938d-987b043ae4a2" Nov 04 05:00:21.409779 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:21.409752 4395 scope.go:122] "RemoveContainer" containerID="9f26d9f945d84c47138eec55fdf959460744264bf807d05f04eb8a1852216ff3" Nov 04 05:00:22.390216 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:22.390170 4395 generic.go:356] "Generic (PLEG): container finished" podID="eeb6bcee-a132-4152-aea6-7164e9a6e003" containerID="76027d05e0e6d5c13bf467ccb91b92f8535f085dab46e2639c9142c7d7358520" exitCode=60 Nov 04 05:00:22.390389 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:22.390256 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-db8fae-29896140-sp67z" event={"ID":"eeb6bcee-a132-4152-aea6-7164e9a6e003","Type":"ContainerDied","Data":"76027d05e0e6d5c13bf467ccb91b92f8535f085dab46e2639c9142c7d7358520"} Nov 04 05:00:22.390389 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:22.390344 4395 scope.go:122] "RemoveContainer" containerID="9f26d9f945d84c47138eec55fdf959460744264bf807d05f04eb8a1852216ff3" Nov 04 05:00:22.390688 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:22.390666 4395 scope.go:122] "RemoveContainer" containerID="76027d05e0e6d5c13bf467ccb91b92f8535f085dab46e2639c9142c7d7358520" Nov 04 05:00:22.390948 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:00:22.390922 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-db8fae-29896140-sp67z_odh-ai-gateway-infra(eeb6bcee-a132-4152-aea6-7164e9a6e003)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-db8fae-29896140-sp67z" podUID="eeb6bcee-a132-4152-aea6-7164e9a6e003" Nov 04 05:00:23.409545 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:23.409518 4395 scope.go:122] "RemoveContainer" containerID="70003c3bd44c6a80f2785bc5e6628a2ac462e2abc65819fd634a3ac7940312c1" Nov 04 05:00:23.511742 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:23.511716 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-db8fae-29896140-sp67z"] Nov 04 05:00:23.655956 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:23.655932 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-db8fae-29896140-sp67z" Nov 04 05:00:23.781769 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:23.781695 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeb6bcee-a132-4152-aea6-7164e9a6e003-kube-api-access-j7pwf\" (UniqueName: \"kubernetes.io/projected/eeb6bcee-a132-4152-aea6-7164e9a6e003-kube-api-access-j7pwf\") pod \"eeb6bcee-a132-4152-aea6-7164e9a6e003\" (UID: \"eeb6bcee-a132-4152-aea6-7164e9a6e003\") " Nov 04 05:00:23.781769 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:23.781753 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eeb6bcee-a132-4152-aea6-7164e9a6e003-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb6bcee-a132-4152-aea6-7164e9a6e003-maas-api-service-ca\") pod \"eeb6bcee-a132-4152-aea6-7164e9a6e003\" (UID: \"eeb6bcee-a132-4152-aea6-7164e9a6e003\") " Nov 04 05:00:23.784322 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:23.782564 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeb6bcee-a132-4152-aea6-7164e9a6e003-maas-api-service-ca" pod "eeb6bcee-a132-4152-aea6-7164e9a6e003" (UID: "eeb6bcee-a132-4152-aea6-7164e9a6e003"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:00:23.786685 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:23.786661 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeb6bcee-a132-4152-aea6-7164e9a6e003-kube-api-access-j7pwf" pod "eeb6bcee-a132-4152-aea6-7164e9a6e003" (UID: "eeb6bcee-a132-4152-aea6-7164e9a6e003"). InnerVolumeSpecName "kube-api-access-j7pwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:00:23.883681 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:23.883660 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7pwf\" (UniqueName: \"kubernetes.io/projected/eeb6bcee-a132-4152-aea6-7164e9a6e003-kube-api-access-j7pwf\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:00:23.883681 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:23.883682 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb6bcee-a132-4152-aea6-7164e9a6e003-maas-api-service-ca\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:00:24.405555 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:24.405526 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-db8fae-29896140-sp67z" Nov 04 05:00:24.405684 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:24.405555 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-db8fae-29896140-sp67z" event={"ID":"eeb6bcee-a132-4152-aea6-7164e9a6e003","Type":"ContainerDied","Data":"099a3e13fe41adc6ae21294ffc4aa61bda4b7ab9cbdd27eb82b8545b01fcde4b"} Nov 04 05:00:24.405684 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:24.405612 4395 scope.go:122] "RemoveContainer" containerID="76027d05e0e6d5c13bf467ccb91b92f8535f085dab46e2639c9142c7d7358520" Nov 04 05:00:24.407635 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:24.407614 4395 generic.go:356] "Generic (PLEG): container finished" podID="211196ed-10bb-4595-a811-0151d6b13889" containerID="ca6977ba1721fda48b7b61128f119dd794b73e3e1560bc1d21685f13a84ca014" exitCode=60 Nov 04 05:00:24.407729 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:24.407664 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c5bb6d-29896140-nk784" event={"ID":"211196ed-10bb-4595-a811-0151d6b13889","Type":"ContainerDied","Data":"ca6977ba1721fda48b7b61128f119dd794b73e3e1560bc1d21685f13a84ca014"} Nov 04 05:00:24.408049 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:24.408031 4395 scope.go:122] "RemoveContainer" containerID="ca6977ba1721fda48b7b61128f119dd794b73e3e1560bc1d21685f13a84ca014" Nov 04 05:00:24.408349 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:00:24.408321 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c5bb6d-29896140-nk784_odh-ai-gateway-infra(211196ed-10bb-4595-a811-0151d6b13889)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c5bb6d-29896140-nk784" podUID="211196ed-10bb-4595-a811-0151d6b13889" Nov 04 05:00:24.412599 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:24.409349 4395 scope.go:122] "RemoveContainer" containerID="f614aab3217c852b986e90138950633559089943b7bbeb4e7c719eb2feb31ac7" Nov 04 05:00:24.416611 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:24.416545 4395 scope.go:122] "RemoveContainer" containerID="70003c3bd44c6a80f2785bc5e6628a2ac462e2abc65819fd634a3ac7940312c1" Nov 04 05:00:24.558815 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:24.558785 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-db8fae-29896140-sp67z"] Nov 04 05:00:24.562811 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:24.562791 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-db8fae-29896140-sp67z"] Nov 04 05:00:25.416341 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:25.416317 4395 generic.go:356] "Generic (PLEG): container finished" podID="ae07bf50-a836-4192-85a4-ebee4dd2a3af" containerID="3101755e04cbd3c14074b5dcc138461d2b55c24e882103c9ec53402310207239" exitCode=60 Nov 04 05:00:25.416739 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:25.416394 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-204e38-29896140-vd5n7" event={"ID":"ae07bf50-a836-4192-85a4-ebee4dd2a3af","Type":"ContainerDied","Data":"3101755e04cbd3c14074b5dcc138461d2b55c24e882103c9ec53402310207239"} Nov 04 05:00:25.416739 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:25.416443 4395 scope.go:122] "RemoveContainer" containerID="f614aab3217c852b986e90138950633559089943b7bbeb4e7c719eb2feb31ac7" Nov 04 05:00:25.416739 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:25.416699 4395 scope.go:122] "RemoveContainer" containerID="3101755e04cbd3c14074b5dcc138461d2b55c24e882103c9ec53402310207239" Nov 04 05:00:25.416942 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:00:25.416920 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-204e38-29896140-vd5n7_odh-ai-gateway-infra(ae07bf50-a836-4192-85a4-ebee4dd2a3af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-204e38-29896140-vd5n7" podUID="ae07bf50-a836-4192-85a4-ebee4dd2a3af" Nov 04 05:00:25.679511 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:25.679483 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c5bb6d-29896140-nk784"] Nov 04 05:00:25.826945 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:25.826925 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c5bb6d-29896140-nk784" Nov 04 05:00:25.905294 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:25.905262 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/211196ed-10bb-4595-a811-0151d6b13889-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/211196ed-10bb-4595-a811-0151d6b13889-maas-api-service-ca\") pod \"211196ed-10bb-4595-a811-0151d6b13889\" (UID: \"211196ed-10bb-4595-a811-0151d6b13889\") " Nov 04 05:00:25.905448 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:25.905420 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/211196ed-10bb-4595-a811-0151d6b13889-kube-api-access-xdsxs\" (UniqueName: \"kubernetes.io/projected/211196ed-10bb-4595-a811-0151d6b13889-kube-api-access-xdsxs\") pod \"211196ed-10bb-4595-a811-0151d6b13889\" (UID: \"211196ed-10bb-4595-a811-0151d6b13889\") " Nov 04 05:00:25.905596 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:25.905575 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/211196ed-10bb-4595-a811-0151d6b13889-maas-api-service-ca" pod "211196ed-10bb-4595-a811-0151d6b13889" (UID: "211196ed-10bb-4595-a811-0151d6b13889"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:00:25.905784 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:25.905768 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/211196ed-10bb-4595-a811-0151d6b13889-maas-api-service-ca\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:00:25.907589 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:25.907567 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/211196ed-10bb-4595-a811-0151d6b13889-kube-api-access-xdsxs" pod "211196ed-10bb-4595-a811-0151d6b13889" (UID: "211196ed-10bb-4595-a811-0151d6b13889"). InnerVolumeSpecName "kube-api-access-xdsxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:00:26.006989 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:26.006929 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdsxs\" (UniqueName: \"kubernetes.io/projected/211196ed-10bb-4595-a811-0151d6b13889-kube-api-access-xdsxs\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:00:26.409849 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:26.409815 4395 scope.go:122] "RemoveContainer" containerID="4a5597ed158f79a7e8878c6770d022a72445e324e50120a2260bf32172187bfd" Nov 04 05:00:26.417872 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:26.417850 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eeb6bcee-a132-4152-aea6-7164e9a6e003" path="/var/lib/kubelet/pods/eeb6bcee-a132-4152-aea6-7164e9a6e003/volumes" Nov 04 05:00:26.429770 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:26.428098 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c5bb6d-29896140-nk784" event={"ID":"211196ed-10bb-4595-a811-0151d6b13889","Type":"ContainerDied","Data":"84b8aa6b297aea22739378d4987f74ae512606d67d594af53709fac9fd0b6c76"} Nov 04 05:00:26.429770 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:26.428126 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c5bb6d-29896140-nk784" Nov 04 05:00:26.429770 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:26.428150 4395 scope.go:122] "RemoveContainer" containerID="ca6977ba1721fda48b7b61128f119dd794b73e3e1560bc1d21685f13a84ca014" Nov 04 05:00:26.535343 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:26.535313 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-204e38-29896140-vd5n7"] Nov 04 05:00:26.581640 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:26.581614 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c5bb6d-29896140-nk784"] Nov 04 05:00:26.585887 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:26.585865 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c5bb6d-29896140-nk784"] Nov 04 05:00:26.672748 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:26.672727 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-204e38-29896140-vd5n7" Nov 04 05:00:26.816339 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:26.816286 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae07bf50-a836-4192-85a4-ebee4dd2a3af-kube-api-access-sv625\" (UniqueName: \"kubernetes.io/projected/ae07bf50-a836-4192-85a4-ebee4dd2a3af-kube-api-access-sv625\") pod \"ae07bf50-a836-4192-85a4-ebee4dd2a3af\" (UID: \"ae07bf50-a836-4192-85a4-ebee4dd2a3af\") " Nov 04 05:00:26.816510 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:26.816409 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae07bf50-a836-4192-85a4-ebee4dd2a3af-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae07bf50-a836-4192-85a4-ebee4dd2a3af-maas-api-service-ca\") pod \"ae07bf50-a836-4192-85a4-ebee4dd2a3af\" (UID: \"ae07bf50-a836-4192-85a4-ebee4dd2a3af\") " Nov 04 05:00:26.816753 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:26.816728 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae07bf50-a836-4192-85a4-ebee4dd2a3af-maas-api-service-ca" pod "ae07bf50-a836-4192-85a4-ebee4dd2a3af" (UID: "ae07bf50-a836-4192-85a4-ebee4dd2a3af"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:00:26.816896 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:26.816877 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae07bf50-a836-4192-85a4-ebee4dd2a3af-maas-api-service-ca\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:00:26.818834 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:26.818812 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae07bf50-a836-4192-85a4-ebee4dd2a3af-kube-api-access-sv625" pod "ae07bf50-a836-4192-85a4-ebee4dd2a3af" (UID: "ae07bf50-a836-4192-85a4-ebee4dd2a3af"). InnerVolumeSpecName "kube-api-access-sv625". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:00:26.917537 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:26.917463 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv625\" (UniqueName: \"kubernetes.io/projected/ae07bf50-a836-4192-85a4-ebee4dd2a3af-kube-api-access-sv625\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:00:27.435747 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:27.435673 4395 generic.go:356] "Generic (PLEG): container finished" podID="aa369b32-590d-4433-938d-987b043ae4a2" containerID="e9e6b7113d68e0b662842fd932583df5dc3fcf621a46e8cc880aa77509e097ec" exitCode=60 Nov 04 05:00:27.436176 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:27.435743 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-204fa0-29896140-7xcmh" event={"ID":"aa369b32-590d-4433-938d-987b043ae4a2","Type":"ContainerDied","Data":"e9e6b7113d68e0b662842fd932583df5dc3fcf621a46e8cc880aa77509e097ec"} Nov 04 05:00:27.436176 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:27.435802 4395 scope.go:122] "RemoveContainer" containerID="4a5597ed158f79a7e8878c6770d022a72445e324e50120a2260bf32172187bfd" Nov 04 05:00:27.436176 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:27.436123 4395 scope.go:122] "RemoveContainer" containerID="e9e6b7113d68e0b662842fd932583df5dc3fcf621a46e8cc880aa77509e097ec" Nov 04 05:00:27.436424 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:00:27.436404 4395 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-204fa0-29896140-7xcmh_odh-ai-gateway-infra(aa369b32-590d-4433-938d-987b043ae4a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-204fa0-29896140-7xcmh" podUID="aa369b32-590d-4433-938d-987b043ae4a2" Nov 04 05:00:27.437518 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:27.437494 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-204e38-29896140-vd5n7" Nov 04 05:00:27.437646 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:27.437515 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-204e38-29896140-vd5n7" event={"ID":"ae07bf50-a836-4192-85a4-ebee4dd2a3af","Type":"ContainerDied","Data":"574306ca8f50e40278ea37cbfbe1bf39b69d49bfb67c37ed20a8a10c3dca7447"} Nov 04 05:00:27.449032 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:27.449015 4395 scope.go:122] "RemoveContainer" containerID="3101755e04cbd3c14074b5dcc138461d2b55c24e882103c9ec53402310207239" Nov 04 05:00:27.648548 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:27.648525 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-204e38-29896140-vd5n7"] Nov 04 05:00:27.654784 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:27.654760 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-204e38-29896140-vd5n7"] Nov 04 05:00:28.416602 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:28.416569 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="211196ed-10bb-4595-a811-0151d6b13889" path="/var/lib/kubelet/pods/211196ed-10bb-4595-a811-0151d6b13889/volumes" Nov 04 05:00:28.417050 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:28.417027 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae07bf50-a836-4192-85a4-ebee4dd2a3af" path="/var/lib/kubelet/pods/ae07bf50-a836-4192-85a4-ebee4dd2a3af/volumes" Nov 04 05:00:28.555813 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:28.555788 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-204fa0-29896140-7xcmh"] Nov 04 05:00:28.704183 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:28.704163 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-204fa0-29896140-7xcmh" Nov 04 05:00:28.837076 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:28.837044 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa369b32-590d-4433-938d-987b043ae4a2-kube-api-access-cr9ll\" (UniqueName: \"kubernetes.io/projected/aa369b32-590d-4433-938d-987b043ae4a2-kube-api-access-cr9ll\") pod \"aa369b32-590d-4433-938d-987b043ae4a2\" (UID: \"aa369b32-590d-4433-938d-987b043ae4a2\") " Nov 04 05:00:28.837372 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:28.837140 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa369b32-590d-4433-938d-987b043ae4a2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa369b32-590d-4433-938d-987b043ae4a2-maas-api-service-ca\") pod \"aa369b32-590d-4433-938d-987b043ae4a2\" (UID: \"aa369b32-590d-4433-938d-987b043ae4a2\") " Nov 04 05:00:28.837607 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:28.837583 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa369b32-590d-4433-938d-987b043ae4a2-maas-api-service-ca" pod "aa369b32-590d-4433-938d-987b043ae4a2" (UID: "aa369b32-590d-4433-938d-987b043ae4a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:00:28.839986 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:28.839955 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa369b32-590d-4433-938d-987b043ae4a2-kube-api-access-cr9ll" pod "aa369b32-590d-4433-938d-987b043ae4a2" (UID: "aa369b32-590d-4433-938d-987b043ae4a2"). InnerVolumeSpecName "kube-api-access-cr9ll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:00:28.938706 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:28.938638 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr9ll\" (UniqueName: \"kubernetes.io/projected/aa369b32-590d-4433-938d-987b043ae4a2-kube-api-access-cr9ll\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:00:28.938706 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:28.938670 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa369b32-590d-4433-938d-987b043ae4a2-maas-api-service-ca\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:00:29.462279 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:29.462249 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-204fa0-29896140-7xcmh" Nov 04 05:00:29.462435 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:29.462275 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-204fa0-29896140-7xcmh" event={"ID":"aa369b32-590d-4433-938d-987b043ae4a2","Type":"ContainerDied","Data":"5a9efbb0a26fa64335ea361a8dc733d739decf77fd0d54f3785eed4b740ba1bb"} Nov 04 05:00:29.462435 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:29.462373 4395 scope.go:122] "RemoveContainer" containerID="e9e6b7113d68e0b662842fd932583df5dc3fcf621a46e8cc880aa77509e097ec" Nov 04 05:00:29.615576 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:29.615548 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-204fa0-29896140-7xcmh"] Nov 04 05:00:29.661385 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:29.661356 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-204fa0-29896140-7xcmh"] Nov 04 05:00:30.415769 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:30.415735 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa369b32-590d-4433-938d-987b043ae4a2" path="/var/lib/kubelet/pods/aa369b32-590d-4433-938d-987b043ae4a2/volumes" Nov 04 05:00:33.461371 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:00:33.461335 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 05:00:39.222555 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:39.222523 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz"] Nov 04 05:00:39.223056 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:39.222854 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" podUID="fbd01e49-964f-4c3f-a14b-249e4047fe34" containerName="istio-proxy" containerID="cri-o://89fae1e93f81d96e91a56b87a85ad5734cf30077bc0dd4c8c6af0c0b273d38a7" gracePeriod=30 Nov 04 05:00:39.268937 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:39.268905 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl"] Nov 04 05:00:39.269150 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:39.269114 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" podUID="115eb9f0-c97e-4133-939f-1797a4157820" containerName="istio-proxy" containerID="cri-o://b637ab3eec393845c57f37ef4a549aa08f5ce88f4b775f2ac809b7f2703786ec" gracePeriod=30 Nov 04 05:00:41.066710 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:41.066672 4395 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-qrtfl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 05:00:41.067093 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:41.066747 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" podUID="115eb9f0-c97e-4133-939f-1797a4157820" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 05:00:44.539166 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.539141 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" Nov 04 05:00:44.558547 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.558519 4395 generic.go:356] "Generic (PLEG): container finished" podID="fbd01e49-964f-4c3f-a14b-249e4047fe34" containerID="89fae1e93f81d96e91a56b87a85ad5734cf30077bc0dd4c8c6af0c0b273d38a7" exitCode=0 Nov 04 05:00:44.558690 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.558583 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" Nov 04 05:00:44.558690 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.558596 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" event={"ID":"fbd01e49-964f-4c3f-a14b-249e4047fe34","Type":"ContainerDied","Data":"89fae1e93f81d96e91a56b87a85ad5734cf30077bc0dd4c8c6af0c0b273d38a7"} Nov 04 05:00:44.558690 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.558656 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz" event={"ID":"fbd01e49-964f-4c3f-a14b-249e4047fe34","Type":"ContainerDied","Data":"95513cfd089cb7e9de0eeb4adacec12ba00ae6b61ca748b33cc86e909e263f48"} Nov 04 05:00:44.558690 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.558674 4395 scope.go:122] "RemoveContainer" containerID="89fae1e93f81d96e91a56b87a85ad5734cf30077bc0dd4c8c6af0c0b273d38a7" Nov 04 05:00:44.560363 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.560342 4395 generic.go:356] "Generic (PLEG): container finished" podID="115eb9f0-c97e-4133-939f-1797a4157820" containerID="b637ab3eec393845c57f37ef4a549aa08f5ce88f4b775f2ac809b7f2703786ec" exitCode=0 Nov 04 05:00:44.560484 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.560428 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" event={"ID":"115eb9f0-c97e-4133-939f-1797a4157820","Type":"ContainerDied","Data":"b637ab3eec393845c57f37ef4a549aa08f5ce88f4b775f2ac809b7f2703786ec"} Nov 04 05:00:44.560551 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.560490 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" event={"ID":"115eb9f0-c97e-4133-939f-1797a4157820","Type":"ContainerDied","Data":"757c5c56566d7b55b19d145c38f06bd79aeb8d99fc8074103805a7171691e25b"} Nov 04 05:00:44.560551 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.560514 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="757c5c56566d7b55b19d145c38f06bd79aeb8d99fc8074103805a7171691e25b" Nov 04 05:00:44.566091 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.566071 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" Nov 04 05:00:44.568024 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.568001 4395 scope.go:122] "RemoveContainer" containerID="89fae1e93f81d96e91a56b87a85ad5734cf30077bc0dd4c8c6af0c0b273d38a7" Nov 04 05:00:44.568312 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:00:44.568282 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89fae1e93f81d96e91a56b87a85ad5734cf30077bc0dd4c8c6af0c0b273d38a7\": container with ID starting with 89fae1e93f81d96e91a56b87a85ad5734cf30077bc0dd4c8c6af0c0b273d38a7 not found: ID does not exist" containerID="89fae1e93f81d96e91a56b87a85ad5734cf30077bc0dd4c8c6af0c0b273d38a7" Nov 04 05:00:44.568422 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.568329 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89fae1e93f81d96e91a56b87a85ad5734cf30077bc0dd4c8c6af0c0b273d38a7"} err="failed to get container status \"89fae1e93f81d96e91a56b87a85ad5734cf30077bc0dd4c8c6af0c0b273d38a7\": rpc error: code = NotFound desc = could not find container \"89fae1e93f81d96e91a56b87a85ad5734cf30077bc0dd4c8c6af0c0b273d38a7\": container with ID starting with 89fae1e93f81d96e91a56b87a85ad5734cf30077bc0dd4c8c6af0c0b273d38a7 not found: ID does not exist" Nov 04 05:00:44.568422 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.568346 4395 scope.go:122] "RemoveContainer" containerID="5346ce7606aa23646860e1c24f3939bf7231b3d84c1c22054764b003337b0440" Nov 04 05:00:44.608784 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.608756 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115eb9f0-c97e-4133-939f-1797a4157820-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115eb9f0-c97e-4133-939f-1797a4157820-workload-socket\") pod \"115eb9f0-c97e-4133-939f-1797a4157820\" (UID: \"115eb9f0-c97e-4133-939f-1797a4157820\") " Nov 04 05:00:44.608868 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.608789 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbd01e49-964f-4c3f-a14b-249e4047fe34-istio-token\" (UniqueName: \"kubernetes.io/projected/fbd01e49-964f-4c3f-a14b-249e4047fe34-istio-token\") pod \"fbd01e49-964f-4c3f-a14b-249e4047fe34\" (UID: \"fbd01e49-964f-4c3f-a14b-249e4047fe34\") " Nov 04 05:00:44.608868 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.608824 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fbd01e49-964f-4c3f-a14b-249e4047fe34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbd01e49-964f-4c3f-a14b-249e4047fe34-istio-podinfo\") pod \"fbd01e49-964f-4c3f-a14b-249e4047fe34\" (UID: \"fbd01e49-964f-4c3f-a14b-249e4047fe34\") " Nov 04 05:00:44.608979 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.608955 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbd01e49-964f-4c3f-a14b-249e4047fe34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd01e49-964f-4c3f-a14b-249e4047fe34-credential-socket\") pod \"fbd01e49-964f-4c3f-a14b-249e4047fe34\" (UID: \"fbd01e49-964f-4c3f-a14b-249e4047fe34\") " Nov 04 05:00:44.609037 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.608999 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbd01e49-964f-4c3f-a14b-249e4047fe34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbd01e49-964f-4c3f-a14b-249e4047fe34-workload-certs\") pod \"fbd01e49-964f-4c3f-a14b-249e4047fe34\" (UID: \"fbd01e49-964f-4c3f-a14b-249e4047fe34\") " Nov 04 05:00:44.609093 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.609036 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbd01e49-964f-4c3f-a14b-249e4047fe34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbd01e49-964f-4c3f-a14b-249e4047fe34-istio-envoy\") pod \"fbd01e49-964f-4c3f-a14b-249e4047fe34\" (UID: \"fbd01e49-964f-4c3f-a14b-249e4047fe34\") " Nov 04 05:00:44.609093 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.609045 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115eb9f0-c97e-4133-939f-1797a4157820-workload-socket" pod "115eb9f0-c97e-4133-939f-1797a4157820" (UID: "115eb9f0-c97e-4133-939f-1797a4157820"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:00:44.609093 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.609086 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/115eb9f0-c97e-4133-939f-1797a4157820-istio-token\" (UniqueName: \"kubernetes.io/projected/115eb9f0-c97e-4133-939f-1797a4157820-istio-token\") pod \"115eb9f0-c97e-4133-939f-1797a4157820\" (UID: \"115eb9f0-c97e-4133-939f-1797a4157820\") " Nov 04 05:00:44.609243 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.609135 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbd01e49-964f-4c3f-a14b-249e4047fe34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd01e49-964f-4c3f-a14b-249e4047fe34-workload-socket\") pod \"fbd01e49-964f-4c3f-a14b-249e4047fe34\" (UID: \"fbd01e49-964f-4c3f-a14b-249e4047fe34\") " Nov 04 05:00:44.609243 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.609196 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbd01e49-964f-4c3f-a14b-249e4047fe34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbd01e49-964f-4c3f-a14b-249e4047fe34-istio-data\") pod \"fbd01e49-964f-4c3f-a14b-249e4047fe34\" (UID: \"fbd01e49-964f-4c3f-a14b-249e4047fe34\") " Nov 04 05:00:44.609243 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.609236 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115eb9f0-c97e-4133-939f-1797a4157820-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115eb9f0-c97e-4133-939f-1797a4157820-workload-certs\") pod \"115eb9f0-c97e-4133-939f-1797a4157820\" (UID: \"115eb9f0-c97e-4133-939f-1797a4157820\") " Nov 04 05:00:44.609435 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.609257 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd01e49-964f-4c3f-a14b-249e4047fe34-workload-certs" pod "fbd01e49-964f-4c3f-a14b-249e4047fe34" (UID: "fbd01e49-964f-4c3f-a14b-249e4047fe34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:00:44.609435 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.609282 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115eb9f0-c97e-4133-939f-1797a4157820-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115eb9f0-c97e-4133-939f-1797a4157820-credential-socket\") pod \"115eb9f0-c97e-4133-939f-1797a4157820\" (UID: \"115eb9f0-c97e-4133-939f-1797a4157820\") " Nov 04 05:00:44.609435 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.609357 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/115eb9f0-c97e-4133-939f-1797a4157820-kube-api-access-hzzw8\" (UniqueName: \"kubernetes.io/projected/115eb9f0-c97e-4133-939f-1797a4157820-kube-api-access-hzzw8\") pod \"115eb9f0-c97e-4133-939f-1797a4157820\" (UID: \"115eb9f0-c97e-4133-939f-1797a4157820\") " Nov 04 05:00:44.609435 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.609401 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115eb9f0-c97e-4133-939f-1797a4157820-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115eb9f0-c97e-4133-939f-1797a4157820-istio-data\") pod \"115eb9f0-c97e-4133-939f-1797a4157820\" (UID: \"115eb9f0-c97e-4133-939f-1797a4157820\") " Nov 04 05:00:44.609629 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.609433 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/115eb9f0-c97e-4133-939f-1797a4157820-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115eb9f0-c97e-4133-939f-1797a4157820-istio-envoy\") pod \"115eb9f0-c97e-4133-939f-1797a4157820\" (UID: \"115eb9f0-c97e-4133-939f-1797a4157820\") " Nov 04 05:00:44.609629 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.609498 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd01e49-964f-4c3f-a14b-249e4047fe34-credential-socket" pod "fbd01e49-964f-4c3f-a14b-249e4047fe34" (UID: "fbd01e49-964f-4c3f-a14b-249e4047fe34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:00:44.609629 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.609511 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbd01e49-964f-4c3f-a14b-249e4047fe34-kube-api-access-xnpvc\" (UniqueName: \"kubernetes.io/projected/fbd01e49-964f-4c3f-a14b-249e4047fe34-kube-api-access-xnpvc\") pod \"fbd01e49-964f-4c3f-a14b-249e4047fe34\" (UID: \"fbd01e49-964f-4c3f-a14b-249e4047fe34\") " Nov 04 05:00:44.609629 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.609554 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/115eb9f0-c97e-4133-939f-1797a4157820-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115eb9f0-c97e-4133-939f-1797a4157820-istiod-ca-cert\") pod \"115eb9f0-c97e-4133-939f-1797a4157820\" (UID: \"115eb9f0-c97e-4133-939f-1797a4157820\") " Nov 04 05:00:44.609629 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.609598 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/115eb9f0-c97e-4133-939f-1797a4157820-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115eb9f0-c97e-4133-939f-1797a4157820-istio-podinfo\") pod \"115eb9f0-c97e-4133-939f-1797a4157820\" (UID: \"115eb9f0-c97e-4133-939f-1797a4157820\") " Nov 04 05:00:44.609904 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.609628 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbd01e49-964f-4c3f-a14b-249e4047fe34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbd01e49-964f-4c3f-a14b-249e4047fe34-istiod-ca-cert\") pod \"fbd01e49-964f-4c3f-a14b-249e4047fe34\" (UID: \"fbd01e49-964f-4c3f-a14b-249e4047fe34\") " Nov 04 05:00:44.610419 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.610086 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/115eb9f0-c97e-4133-939f-1797a4157820-workload-socket\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:00:44.610419 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.610112 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd01e49-964f-4c3f-a14b-249e4047fe34-credential-socket\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:00:44.610419 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.610128 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbd01e49-964f-4c3f-a14b-249e4047fe34-workload-certs\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:00:44.610419 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.609510 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd01e49-964f-4c3f-a14b-249e4047fe34-istio-data" pod "fbd01e49-964f-4c3f-a14b-249e4047fe34" (UID: "fbd01e49-964f-4c3f-a14b-249e4047fe34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:00:44.610419 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.609688 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd01e49-964f-4c3f-a14b-249e4047fe34-workload-socket" pod "fbd01e49-964f-4c3f-a14b-249e4047fe34" (UID: "fbd01e49-964f-4c3f-a14b-249e4047fe34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:00:44.610419 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.610084 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115eb9f0-c97e-4133-939f-1797a4157820-workload-certs" pod "115eb9f0-c97e-4133-939f-1797a4157820" (UID: "115eb9f0-c97e-4133-939f-1797a4157820"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:00:44.610419 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.610192 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115eb9f0-c97e-4133-939f-1797a4157820-istio-data" pod "115eb9f0-c97e-4133-939f-1797a4157820" (UID: "115eb9f0-c97e-4133-939f-1797a4157820"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:00:44.610419 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.610230 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/115eb9f0-c97e-4133-939f-1797a4157820-istiod-ca-cert" pod "115eb9f0-c97e-4133-939f-1797a4157820" (UID: "115eb9f0-c97e-4133-939f-1797a4157820"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:00:44.610419 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.610392 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115eb9f0-c97e-4133-939f-1797a4157820-credential-socket" pod "115eb9f0-c97e-4133-939f-1797a4157820" (UID: "115eb9f0-c97e-4133-939f-1797a4157820"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:00:44.611004 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.610459 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbd01e49-964f-4c3f-a14b-249e4047fe34-istiod-ca-cert" pod "fbd01e49-964f-4c3f-a14b-249e4047fe34" (UID: "fbd01e49-964f-4c3f-a14b-249e4047fe34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:00:44.611504 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.611476 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd01e49-964f-4c3f-a14b-249e4047fe34-istio-token" pod "fbd01e49-964f-4c3f-a14b-249e4047fe34" (UID: "fbd01e49-964f-4c3f-a14b-249e4047fe34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:00:44.611668 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.611639 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd01e49-964f-4c3f-a14b-249e4047fe34-istio-envoy" pod "fbd01e49-964f-4c3f-a14b-249e4047fe34" (UID: "fbd01e49-964f-4c3f-a14b-249e4047fe34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:00:44.611898 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.611879 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd01e49-964f-4c3f-a14b-249e4047fe34-kube-api-access-xnpvc" pod "fbd01e49-964f-4c3f-a14b-249e4047fe34" (UID: "fbd01e49-964f-4c3f-a14b-249e4047fe34"). InnerVolumeSpecName "kube-api-access-xnpvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:00:44.613243 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.613217 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbd01e49-964f-4c3f-a14b-249e4047fe34-istio-podinfo" pod "fbd01e49-964f-4c3f-a14b-249e4047fe34" (UID: "fbd01e49-964f-4c3f-a14b-249e4047fe34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:00:44.613361 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.613247 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115eb9f0-c97e-4133-939f-1797a4157820-istio-envoy" pod "115eb9f0-c97e-4133-939f-1797a4157820" (UID: "115eb9f0-c97e-4133-939f-1797a4157820"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:00:44.613361 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.613254 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/115eb9f0-c97e-4133-939f-1797a4157820-istio-podinfo" pod "115eb9f0-c97e-4133-939f-1797a4157820" (UID: "115eb9f0-c97e-4133-939f-1797a4157820"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:00:44.613361 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.613308 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/115eb9f0-c97e-4133-939f-1797a4157820-kube-api-access-hzzw8" pod "115eb9f0-c97e-4133-939f-1797a4157820" (UID: "115eb9f0-c97e-4133-939f-1797a4157820"). InnerVolumeSpecName "kube-api-access-hzzw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:00:44.613535 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.613519 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/115eb9f0-c97e-4133-939f-1797a4157820-istio-token" pod "115eb9f0-c97e-4133-939f-1797a4157820" (UID: "115eb9f0-c97e-4133-939f-1797a4157820"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:00:44.711219 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.711192 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnpvc\" (UniqueName: \"kubernetes.io/projected/fbd01e49-964f-4c3f-a14b-249e4047fe34-kube-api-access-xnpvc\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:00:44.711219 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.711219 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/115eb9f0-c97e-4133-939f-1797a4157820-istiod-ca-cert\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:00:44.711410 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.711228 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/115eb9f0-c97e-4133-939f-1797a4157820-istio-podinfo\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:00:44.711410 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.711237 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbd01e49-964f-4c3f-a14b-249e4047fe34-istiod-ca-cert\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:00:44.711410 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.711246 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbd01e49-964f-4c3f-a14b-249e4047fe34-istio-token\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:00:44.711410 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.711254 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbd01e49-964f-4c3f-a14b-249e4047fe34-istio-podinfo\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:00:44.711410 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.711263 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbd01e49-964f-4c3f-a14b-249e4047fe34-istio-envoy\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:00:44.711410 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.711270 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/115eb9f0-c97e-4133-939f-1797a4157820-istio-token\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:00:44.711410 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.711278 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd01e49-964f-4c3f-a14b-249e4047fe34-workload-socket\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:00:44.711410 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.711286 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbd01e49-964f-4c3f-a14b-249e4047fe34-istio-data\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:00:44.711410 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.711307 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/115eb9f0-c97e-4133-939f-1797a4157820-workload-certs\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:00:44.711410 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.711318 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/115eb9f0-c97e-4133-939f-1797a4157820-credential-socket\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:00:44.711410 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.711326 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzzw8\" (UniqueName: \"kubernetes.io/projected/115eb9f0-c97e-4133-939f-1797a4157820-kube-api-access-hzzw8\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:00:44.711410 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.711334 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/115eb9f0-c97e-4133-939f-1797a4157820-istio-data\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:00:44.711410 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.711342 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/115eb9f0-c97e-4133-939f-1797a4157820-istio-envoy\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:00:44.973215 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.973185 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz"] Nov 04 05:00:44.977393 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:44.977366 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cd7gz"] Nov 04 05:00:45.568674 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:45.568643 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl" Nov 04 05:00:45.786742 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:45.786711 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl"] Nov 04 05:00:45.791436 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:45.791408 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qrtfl"] Nov 04 05:00:46.416348 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:46.416319 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="115eb9f0-c97e-4133-939f-1797a4157820" path="/var/lib/kubelet/pods/115eb9f0-c97e-4133-939f-1797a4157820/volumes" Nov 04 05:00:46.416764 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:46.416751 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbd01e49-964f-4c3f-a14b-249e4047fe34" path="/var/lib/kubelet/pods/fbd01e49-964f-4c3f-a14b-249e4047fe34/volumes" Nov 04 05:00:55.247006 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.246976 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7053-ecea19"] Nov 04 05:00:55.247636 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.247615 4395 cpu_manager.go:414] "containerMap: removing container" podUID="eeb6bcee-a132-4152-aea6-7164e9a6e003" containerName="cleanup" Nov 04 05:00:55.247713 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.247639 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeb6bcee-a132-4152-aea6-7164e9a6e003" containerName="cleanup" Nov 04 05:00:55.247713 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.247662 4395 cpu_manager.go:414] "containerMap: removing container" podUID="211196ed-10bb-4595-a811-0151d6b13889" containerName="cleanup" Nov 04 05:00:55.247713 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.247671 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="211196ed-10bb-4595-a811-0151d6b13889" containerName="cleanup" Nov 04 05:00:55.247713 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.247683 4395 cpu_manager.go:414] "containerMap: removing container" podUID="115eb9f0-c97e-4133-939f-1797a4157820" containerName="istio-proxy" Nov 04 05:00:55.247713 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.247690 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="115eb9f0-c97e-4133-939f-1797a4157820" containerName="istio-proxy" Nov 04 05:00:55.247713 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.247704 4395 cpu_manager.go:414] "containerMap: removing container" podUID="fbd01e49-964f-4c3f-a14b-249e4047fe34" containerName="istio-proxy" Nov 04 05:00:55.247713 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.247711 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbd01e49-964f-4c3f-a14b-249e4047fe34" containerName="istio-proxy" Nov 04 05:00:55.247957 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.247734 4395 cpu_manager.go:414] "containerMap: removing container" podUID="ae07bf50-a836-4192-85a4-ebee4dd2a3af" containerName="cleanup" Nov 04 05:00:55.247957 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.247742 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae07bf50-a836-4192-85a4-ebee4dd2a3af" containerName="cleanup" Nov 04 05:00:55.247957 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.247768 4395 cpu_manager.go:414] "containerMap: removing container" podUID="aa369b32-590d-4433-938d-987b043ae4a2" containerName="cleanup" Nov 04 05:00:55.247957 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.247775 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa369b32-590d-4433-938d-987b043ae4a2" containerName="cleanup" Nov 04 05:00:55.247957 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.247785 4395 cpu_manager.go:414] "containerMap: removing container" podUID="aa369b32-590d-4433-938d-987b043ae4a2" containerName="cleanup" Nov 04 05:00:55.247957 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.247791 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa369b32-590d-4433-938d-987b043ae4a2" containerName="cleanup" Nov 04 05:00:55.247957 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.247800 4395 cpu_manager.go:414] "containerMap: removing container" podUID="211196ed-10bb-4595-a811-0151d6b13889" containerName="cleanup" Nov 04 05:00:55.247957 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.247807 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="211196ed-10bb-4595-a811-0151d6b13889" containerName="cleanup" Nov 04 05:00:55.247957 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.247815 4395 cpu_manager.go:414] "containerMap: removing container" podUID="ae07bf50-a836-4192-85a4-ebee4dd2a3af" containerName="cleanup" Nov 04 05:00:55.247957 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.247822 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae07bf50-a836-4192-85a4-ebee4dd2a3af" containerName="cleanup" Nov 04 05:00:55.247957 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.247839 4395 cpu_manager.go:414] "containerMap: removing container" podUID="ae07bf50-a836-4192-85a4-ebee4dd2a3af" containerName="cleanup" Nov 04 05:00:55.247957 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.247845 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae07bf50-a836-4192-85a4-ebee4dd2a3af" containerName="cleanup" Nov 04 05:00:55.247957 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.247856 4395 cpu_manager.go:414] "containerMap: removing container" podUID="aa369b32-590d-4433-938d-987b043ae4a2" containerName="cleanup" Nov 04 05:00:55.247957 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.247863 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa369b32-590d-4433-938d-987b043ae4a2" containerName="cleanup" Nov 04 05:00:55.247957 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.247872 4395 cpu_manager.go:414] "containerMap: removing container" podUID="eeb6bcee-a132-4152-aea6-7164e9a6e003" containerName="cleanup" Nov 04 05:00:55.247957 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.247879 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeb6bcee-a132-4152-aea6-7164e9a6e003" containerName="cleanup" Nov 04 05:00:55.248472 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.247976 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="115eb9f0-c97e-4133-939f-1797a4157820" containerName="istio-proxy" Nov 04 05:00:55.248472 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.247988 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa369b32-590d-4433-938d-987b043ae4a2" containerName="cleanup" Nov 04 05:00:55.248472 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.247998 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa369b32-590d-4433-938d-987b043ae4a2" containerName="cleanup" Nov 04 05:00:55.248472 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.248007 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="211196ed-10bb-4595-a811-0151d6b13889" containerName="cleanup" Nov 04 05:00:55.248472 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.248015 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae07bf50-a836-4192-85a4-ebee4dd2a3af" containerName="cleanup" Nov 04 05:00:55.248472 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.248025 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae07bf50-a836-4192-85a4-ebee4dd2a3af" containerName="cleanup" Nov 04 05:00:55.248472 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.248032 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa369b32-590d-4433-938d-987b043ae4a2" containerName="cleanup" Nov 04 05:00:55.248472 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.248041 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeb6bcee-a132-4152-aea6-7164e9a6e003" containerName="cleanup" Nov 04 05:00:55.248472 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.248051 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeb6bcee-a132-4152-aea6-7164e9a6e003" containerName="cleanup" Nov 04 05:00:55.248472 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.248061 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="211196ed-10bb-4595-a811-0151d6b13889" containerName="cleanup" Nov 04 05:00:55.248472 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.248070 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbd01e49-964f-4c3f-a14b-249e4047fe34" containerName="istio-proxy" Nov 04 05:00:55.248472 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.248079 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="211196ed-10bb-4595-a811-0151d6b13889" containerName="cleanup" Nov 04 05:00:55.248472 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.248088 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeb6bcee-a132-4152-aea6-7164e9a6e003" containerName="cleanup" Nov 04 05:00:55.253552 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.253529 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7053-ecea19" Nov 04 05:00:55.305054 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.305028 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7053-ecea19"] Nov 04 05:00:55.348540 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.348518 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qsq57\"" Nov 04 05:00:55.353824 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.353804 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7041-cccc51"] Nov 04 05:00:55.354197 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.354184 4395 cpu_manager.go:414] "containerMap: removing container" podUID="eeb6bcee-a132-4152-aea6-7164e9a6e003" containerName="cleanup" Nov 04 05:00:55.354236 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.354198 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeb6bcee-a132-4152-aea6-7164e9a6e003" containerName="cleanup" Nov 04 05:00:55.354236 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.354212 4395 cpu_manager.go:414] "containerMap: removing container" podUID="211196ed-10bb-4595-a811-0151d6b13889" containerName="cleanup" Nov 04 05:00:55.354236 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.354217 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="211196ed-10bb-4595-a811-0151d6b13889" containerName="cleanup" Nov 04 05:00:55.354236 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.354226 4395 cpu_manager.go:414] "containerMap: removing container" podUID="115eb9f0-c97e-4133-939f-1797a4157820" containerName="istio-proxy" Nov 04 05:00:55.354236 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.354231 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="115eb9f0-c97e-4133-939f-1797a4157820" containerName="istio-proxy" Nov 04 05:00:55.354408 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.354337 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="115eb9f0-c97e-4133-939f-1797a4157820" containerName="istio-proxy" Nov 04 05:00:55.354408 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.354348 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae07bf50-a836-4192-85a4-ebee4dd2a3af" containerName="cleanup" Nov 04 05:00:55.357598 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.357581 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7041-cccc51" Nov 04 05:00:55.412843 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.412820 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7041-cccc51"] Nov 04 05:00:55.418031 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.418008 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs6lt\" (UniqueName: \"kubernetes.io/projected/7d45770b-8c39-4239-b799-32010d6df85b-kube-api-access-hs6lt\") pod \"test-curl-7053-ecea19\" (UID: \"7d45770b-8c39-4239-b799-32010d6df85b\") " pod="opendatahub/test-curl-7053-ecea19" Nov 04 05:00:55.518960 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.518891 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs6lt\" (UniqueName: \"kubernetes.io/projected/7d45770b-8c39-4239-b799-32010d6df85b-kube-api-access-hs6lt\") pod \"test-curl-7053-ecea19\" (UID: \"7d45770b-8c39-4239-b799-32010d6df85b\") " pod="opendatahub/test-curl-7053-ecea19" Nov 04 05:00:55.518960 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.518940 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z498\" (UniqueName: \"kubernetes.io/projected/b0a64840-53b4-456b-8465-b256b64bfe35-kube-api-access-4z498\") pod \"test-curl-7041-cccc51\" (UID: \"b0a64840-53b4-456b-8465-b256b64bfe35\") " pod="opendatahub/test-curl-7041-cccc51" Nov 04 05:00:55.576407 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.576377 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs6lt\" (UniqueName: \"kubernetes.io/projected/7d45770b-8c39-4239-b799-32010d6df85b-kube-api-access-hs6lt\") pod \"test-curl-7053-ecea19\" (UID: \"7d45770b-8c39-4239-b799-32010d6df85b\") " pod="opendatahub/test-curl-7053-ecea19" Nov 04 05:00:55.619593 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.619566 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z498\" (UniqueName: \"kubernetes.io/projected/b0a64840-53b4-456b-8465-b256b64bfe35-kube-api-access-4z498\") pod \"test-curl-7041-cccc51\" (UID: \"b0a64840-53b4-456b-8465-b256b64bfe35\") " pod="opendatahub/test-curl-7041-cccc51" Nov 04 05:00:55.676760 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.676730 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z498\" (UniqueName: \"kubernetes.io/projected/b0a64840-53b4-456b-8465-b256b64bfe35-kube-api-access-4z498\") pod \"test-curl-7041-cccc51\" (UID: \"b0a64840-53b4-456b-8465-b256b64bfe35\") " pod="opendatahub/test-curl-7041-cccc51" Nov 04 05:00:55.865748 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.865712 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7053-ecea19" Nov 04 05:00:55.970604 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:55.970572 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7041-cccc51" Nov 04 05:00:56.241879 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:56.241848 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7053-ecea19"] Nov 04 05:00:56.242840 ip-10-0-134-224 kubenswrapper[4395]: W1104 05:00:56.242812 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d45770b_8c39_4239_b799_32010d6df85b.slice/crio-57804293846c0f6b86f7b11d82d504fea143ed9ae6f2d1f6d7cf99113bdef422 WatchSource:0}: Error finding container 57804293846c0f6b86f7b11d82d504fea143ed9ae6f2d1f6d7cf99113bdef422: Status 404 returned error can't find the container with id 57804293846c0f6b86f7b11d82d504fea143ed9ae6f2d1f6d7cf99113bdef422 Nov 04 05:00:56.344599 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:56.344577 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7041-cccc51"] Nov 04 05:00:56.346875 ip-10-0-134-224 kubenswrapper[4395]: W1104 05:00:56.346849 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0a64840_53b4_456b_8465_b256b64bfe35.slice/crio-70c11ff54c1ec15a870fa2fe3994c598d35e00f3fe2e4cd174fe1a3877e33f28 WatchSource:0}: Error finding container 70c11ff54c1ec15a870fa2fe3994c598d35e00f3fe2e4cd174fe1a3877e33f28: Status 404 returned error can't find the container with id 70c11ff54c1ec15a870fa2fe3994c598d35e00f3fe2e4cd174fe1a3877e33f28 Nov 04 05:00:56.425500 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:56.425462 4395 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","pod211196ed-10bb-4595-a811-0151d6b13889"] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod211196ed-10bb-4595-a811-0151d6b13889] : Timed out while waiting for systemd to remove kubepods-burstable-pod211196ed_10bb_4595_a811_0151d6b13889.slice" Nov 04 05:00:56.425603 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:56.425500 4395 kubelet_pods.go:2797] "Failed to destroy orphaned pod cgroup" podUID="211196ed-10bb-4595-a811-0151d6b13889" err="failed to delete cgroup paths for [kubepods burstable pod211196ed-10bb-4595-a811-0151d6b13889] : unable to destroy cgroup paths for cgroup [kubepods burstable pod211196ed-10bb-4595-a811-0151d6b13889] : Timed out while waiting for systemd to remove kubepods-burstable-pod211196ed_10bb_4595_a811_0151d6b13889.slice" Nov 04 05:00:56.632912 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:56.632882 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7041-cccc51" event={"ID":"b0a64840-53b4-456b-8465-b256b64bfe35","Type":"ContainerStarted","Data":"70c11ff54c1ec15a870fa2fe3994c598d35e00f3fe2e4cd174fe1a3877e33f28"} Nov 04 05:00:56.634055 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:56.634033 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7053-ecea19" event={"ID":"7d45770b-8c39-4239-b799-32010d6df85b","Type":"ContainerStarted","Data":"57804293846c0f6b86f7b11d82d504fea143ed9ae6f2d1f6d7cf99113bdef422"} Nov 04 05:00:57.641137 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:57.641109 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7041-cccc51" event={"ID":"b0a64840-53b4-456b-8465-b256b64bfe35","Type":"ContainerStarted","Data":"e141e13bea2e6707f15ea89922486f5fc61baa12d736073f7a03a3faa53f4804"} Nov 04 05:00:57.749545 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:57.749447 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7041-cccc51" podStartSLOduration=1.651445665 podStartE2EDuration="2.749433431s" totalImagesPullingTime="1.097987766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:00:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:00:57.747129047 +0000 UTC m=+899.437211495" watchObservedRunningTime="2026-11-04 05:00:57.749433431 +0000 UTC m=+899.439515878" Nov 04 05:00:58.555633 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:58.555600 4395 scope.go:122] "RemoveContainer" containerID="b637ab3eec393845c57f37ef4a549aa08f5ce88f4b775f2ac809b7f2703786ec" Nov 04 05:00:58.648511 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:58.648478 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7053-ecea19" event={"ID":"7d45770b-8c39-4239-b799-32010d6df85b","Type":"ContainerStarted","Data":"98d2ea48bc1660190bba9cfd7892574939948529ad271c5493568987598e4be7"} Nov 04 05:00:58.761139 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:58.761086 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7053-ecea19" podStartSLOduration=2.112474186 podStartE2EDuration="3.761074383s" totalImagesPullingTime="1.648600197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:00:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:00:58.76013144 +0000 UTC m=+900.450213889" watchObservedRunningTime="2026-11-04 05:00:58.761074383 +0000 UTC m=+900.451156829" Nov 04 05:00:59.302323 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:59.301356 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-a48f7e/distinct-w4-a48f7e-kserve-8d66794c4-94568"] Nov 04 05:00:59.312322 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:59.312260 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a48f7e/distinct-w4-a48f7e-kserve-8d66794c4-94568" Nov 04 05:00:59.366144 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:59.364749 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-a48f7e/distinct-w4-a48f7e-kserve-8d66794c4-94568"] Nov 04 05:00:59.406224 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:59.406170 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-a48f7e\"/\"kube-root-ca.crt\"" Nov 04 05:00:59.406487 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:59.406288 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-a48f7e\"/\"openshift-service-ca.crt\"" Nov 04 05:00:59.406487 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:59.406417 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-a48f7e\"/\"default-dockercfg-fs466\"" Nov 04 05:00:59.407568 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:59.407542 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-a48f7e\"/\"distinct-w4-a48f7e-kserve-self-signed-certs\"" Nov 04 05:00:59.466972 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:59.466939 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7041-cccc51"] Nov 04 05:00:59.475323 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:59.472742 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61c310e9-1d98-4656-9105-107acf3d4669-model-cache\") pod \"distinct-w4-a48f7e-kserve-8d66794c4-94568\" (UID: \"61c310e9-1d98-4656-9105-107acf3d4669\") " pod="e2e-models-e2e-worker-w4-a48f7e/distinct-w4-a48f7e-kserve-8d66794c4-94568" Nov 04 05:00:59.475323 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:59.472852 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61c310e9-1d98-4656-9105-107acf3d4669-tls-certs\") pod \"distinct-w4-a48f7e-kserve-8d66794c4-94568\" (UID: \"61c310e9-1d98-4656-9105-107acf3d4669\") " pod="e2e-models-e2e-worker-w4-a48f7e/distinct-w4-a48f7e-kserve-8d66794c4-94568" Nov 04 05:00:59.475323 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:59.472925 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9hqc\" (UniqueName: \"kubernetes.io/projected/61c310e9-1d98-4656-9105-107acf3d4669-kube-api-access-l9hqc\") pod \"distinct-w4-a48f7e-kserve-8d66794c4-94568\" (UID: \"61c310e9-1d98-4656-9105-107acf3d4669\") " pod="e2e-models-e2e-worker-w4-a48f7e/distinct-w4-a48f7e-kserve-8d66794c4-94568" Nov 04 05:00:59.475323 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:59.472957 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61c310e9-1d98-4656-9105-107acf3d4669-home\") pod \"distinct-w4-a48f7e-kserve-8d66794c4-94568\" (UID: \"61c310e9-1d98-4656-9105-107acf3d4669\") " pod="e2e-models-e2e-worker-w4-a48f7e/distinct-w4-a48f7e-kserve-8d66794c4-94568" Nov 04 05:00:59.475323 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:59.472991 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61c310e9-1d98-4656-9105-107acf3d4669-dshm\") pod \"distinct-w4-a48f7e-kserve-8d66794c4-94568\" (UID: \"61c310e9-1d98-4656-9105-107acf3d4669\") " pod="e2e-models-e2e-worker-w4-a48f7e/distinct-w4-a48f7e-kserve-8d66794c4-94568" Nov 04 05:00:59.475323 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:59.473017 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61c310e9-1d98-4656-9105-107acf3d4669-tmp-dir\") pod \"distinct-w4-a48f7e-kserve-8d66794c4-94568\" (UID: \"61c310e9-1d98-4656-9105-107acf3d4669\") " pod="e2e-models-e2e-worker-w4-a48f7e/distinct-w4-a48f7e-kserve-8d66794c4-94568" Nov 04 05:00:59.475323 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:59.474522 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7041-cccc51"] Nov 04 05:00:59.574230 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:59.574142 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61c310e9-1d98-4656-9105-107acf3d4669-tls-certs\") pod \"distinct-w4-a48f7e-kserve-8d66794c4-94568\" (UID: \"61c310e9-1d98-4656-9105-107acf3d4669\") " pod="e2e-models-e2e-worker-w4-a48f7e/distinct-w4-a48f7e-kserve-8d66794c4-94568" Nov 04 05:00:59.574230 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:59.574215 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9hqc\" (UniqueName: \"kubernetes.io/projected/61c310e9-1d98-4656-9105-107acf3d4669-kube-api-access-l9hqc\") pod \"distinct-w4-a48f7e-kserve-8d66794c4-94568\" (UID: \"61c310e9-1d98-4656-9105-107acf3d4669\") " pod="e2e-models-e2e-worker-w4-a48f7e/distinct-w4-a48f7e-kserve-8d66794c4-94568" Nov 04 05:00:59.574486 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:59.574249 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61c310e9-1d98-4656-9105-107acf3d4669-home\") pod \"distinct-w4-a48f7e-kserve-8d66794c4-94568\" (UID: \"61c310e9-1d98-4656-9105-107acf3d4669\") " pod="e2e-models-e2e-worker-w4-a48f7e/distinct-w4-a48f7e-kserve-8d66794c4-94568" Nov 04 05:00:59.574486 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:59.574277 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61c310e9-1d98-4656-9105-107acf3d4669-dshm\") pod \"distinct-w4-a48f7e-kserve-8d66794c4-94568\" (UID: \"61c310e9-1d98-4656-9105-107acf3d4669\") " pod="e2e-models-e2e-worker-w4-a48f7e/distinct-w4-a48f7e-kserve-8d66794c4-94568" Nov 04 05:00:59.574486 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:59.574312 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61c310e9-1d98-4656-9105-107acf3d4669-tmp-dir\") pod \"distinct-w4-a48f7e-kserve-8d66794c4-94568\" (UID: \"61c310e9-1d98-4656-9105-107acf3d4669\") " pod="e2e-models-e2e-worker-w4-a48f7e/distinct-w4-a48f7e-kserve-8d66794c4-94568" Nov 04 05:00:59.574486 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:59.574385 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61c310e9-1d98-4656-9105-107acf3d4669-model-cache\") pod \"distinct-w4-a48f7e-kserve-8d66794c4-94568\" (UID: \"61c310e9-1d98-4656-9105-107acf3d4669\") " pod="e2e-models-e2e-worker-w4-a48f7e/distinct-w4-a48f7e-kserve-8d66794c4-94568" Nov 04 05:00:59.574888 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:59.574857 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61c310e9-1d98-4656-9105-107acf3d4669-home\") pod \"distinct-w4-a48f7e-kserve-8d66794c4-94568\" (UID: \"61c310e9-1d98-4656-9105-107acf3d4669\") " pod="e2e-models-e2e-worker-w4-a48f7e/distinct-w4-a48f7e-kserve-8d66794c4-94568" Nov 04 05:00:59.575416 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:59.575387 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61c310e9-1d98-4656-9105-107acf3d4669-model-cache\") pod \"distinct-w4-a48f7e-kserve-8d66794c4-94568\" (UID: \"61c310e9-1d98-4656-9105-107acf3d4669\") " pod="e2e-models-e2e-worker-w4-a48f7e/distinct-w4-a48f7e-kserve-8d66794c4-94568" Nov 04 05:00:59.576237 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:59.576210 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61c310e9-1d98-4656-9105-107acf3d4669-tmp-dir\") pod \"distinct-w4-a48f7e-kserve-8d66794c4-94568\" (UID: \"61c310e9-1d98-4656-9105-107acf3d4669\") " pod="e2e-models-e2e-worker-w4-a48f7e/distinct-w4-a48f7e-kserve-8d66794c4-94568" Nov 04 05:00:59.576939 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:59.576919 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61c310e9-1d98-4656-9105-107acf3d4669-tls-certs\") pod \"distinct-w4-a48f7e-kserve-8d66794c4-94568\" (UID: \"61c310e9-1d98-4656-9105-107acf3d4669\") " pod="e2e-models-e2e-worker-w4-a48f7e/distinct-w4-a48f7e-kserve-8d66794c4-94568" Nov 04 05:00:59.577427 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:59.577401 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61c310e9-1d98-4656-9105-107acf3d4669-dshm\") pod \"distinct-w4-a48f7e-kserve-8d66794c4-94568\" (UID: \"61c310e9-1d98-4656-9105-107acf3d4669\") " pod="e2e-models-e2e-worker-w4-a48f7e/distinct-w4-a48f7e-kserve-8d66794c4-94568" Nov 04 05:00:59.633424 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:59.633384 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9hqc\" (UniqueName: \"kubernetes.io/projected/61c310e9-1d98-4656-9105-107acf3d4669-kube-api-access-l9hqc\") pod \"distinct-w4-a48f7e-kserve-8d66794c4-94568\" (UID: \"61c310e9-1d98-4656-9105-107acf3d4669\") " pod="e2e-models-e2e-worker-w4-a48f7e/distinct-w4-a48f7e-kserve-8d66794c4-94568" Nov 04 05:00:59.654590 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:59.654554 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7041-cccc51" podUID="b0a64840-53b4-456b-8465-b256b64bfe35" containerName="test-curl-7041-cccc51" containerID="cri-o://e141e13bea2e6707f15ea89922486f5fc61baa12d736073f7a03a3faa53f4804" gracePeriod=2 Nov 04 05:00:59.705775 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:00:59.705689 4395 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7041-cccc51\" is forbidden: User \"system:node:ip-10-0-134-224.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-224.us-west-2.compute.internal' and this object" podUID="b0a64840-53b4-456b-8465-b256b64bfe35" pod="opendatahub/test-curl-7041-cccc51" Nov 04 05:00:59.928825 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:00:59.928740 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a48f7e/distinct-w4-a48f7e-kserve-8d66794c4-94568" Nov 04 05:01:00.161310 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:00.161266 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7053-ecea19"] Nov 04 05:01:00.164842 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:00.164817 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7053-ecea19"] Nov 04 05:01:00.332957 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:00.332928 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-a48f7e/distinct-w4-a48f7e-kserve-8d66794c4-94568"] Nov 04 05:01:00.334004 ip-10-0-134-224 kubenswrapper[4395]: W1104 05:01:00.333971 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61c310e9_1d98_4656_9105_107acf3d4669.slice/crio-84579e818ae393393683fb3a2d2fefd5223c67abdf20f484ac5ae9303b330989 WatchSource:0}: Error finding container 84579e818ae393393683fb3a2d2fefd5223c67abdf20f484ac5ae9303b330989: Status 404 returned error can't find the container with id 84579e818ae393393683fb3a2d2fefd5223c67abdf20f484ac5ae9303b330989 Nov 04 05:01:00.662747 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:00.662655 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a48f7e/distinct-w4-a48f7e-kserve-8d66794c4-94568" event={"ID":"61c310e9-1d98-4656-9105-107acf3d4669","Type":"ContainerStarted","Data":"f4d7570710b9fc665411cb9855e692c2434f1f8cd16b2da2e650978d0bd06873"} Nov 04 05:01:00.662747 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:00.662702 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a48f7e/distinct-w4-a48f7e-kserve-8d66794c4-94568" event={"ID":"61c310e9-1d98-4656-9105-107acf3d4669","Type":"ContainerStarted","Data":"84579e818ae393393683fb3a2d2fefd5223c67abdf20f484ac5ae9303b330989"} Nov 04 05:01:00.663258 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:00.662861 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7053-ecea19" podUID="7d45770b-8c39-4239-b799-32010d6df85b" containerName="test-curl-7053-ecea19" containerID="cri-o://98d2ea48bc1660190bba9cfd7892574939948529ad271c5493568987598e4be7" gracePeriod=2 Nov 04 05:01:00.771666 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:00.771615 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-a48f7e/distinct-w4-a48f7e-kserve-8d66794c4-94568" podStartSLOduration=1.771599975 podStartE2EDuration="1.771599975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:01:00.771522186 +0000 UTC m=+902.461604633" watchObservedRunningTime="2026-11-04 05:01:00.771599975 +0000 UTC m=+902.461682422" Nov 04 05:01:00.929362 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:00.929257 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a48f7e/distinct-w4-a48f7e-kserve-8d66794c4-94568" Nov 04 05:01:01.906637 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:01.906611 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7041-cccc51" Nov 04 05:01:02.002067 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:02.001991 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0a64840-53b4-456b-8465-b256b64bfe35-kube-api-access-4z498\" (UniqueName: \"kubernetes.io/projected/b0a64840-53b4-456b-8465-b256b64bfe35-kube-api-access-4z498\") pod \"b0a64840-53b4-456b-8465-b256b64bfe35\" (UID: \"b0a64840-53b4-456b-8465-b256b64bfe35\") " Nov 04 05:01:02.004022 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:02.003995 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0a64840-53b4-456b-8465-b256b64bfe35-kube-api-access-4z498" pod "b0a64840-53b4-456b-8465-b256b64bfe35" (UID: "b0a64840-53b4-456b-8465-b256b64bfe35"). InnerVolumeSpecName "kube-api-access-4z498". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:01:02.102775 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:02.102746 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z498\" (UniqueName: \"kubernetes.io/projected/b0a64840-53b4-456b-8465-b256b64bfe35-kube-api-access-4z498\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:02.416120 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:02.416087 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0a64840-53b4-456b-8465-b256b64bfe35" path="/var/lib/kubelet/pods/b0a64840-53b4-456b-8465-b256b64bfe35/volumes" Nov 04 05:01:02.676955 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:02.676928 4395 generic.go:356] "Generic (PLEG): container finished" podID="b0a64840-53b4-456b-8465-b256b64bfe35" containerID="e141e13bea2e6707f15ea89922486f5fc61baa12d736073f7a03a3faa53f4804" exitCode=137 Nov 04 05:01:02.677067 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:02.676978 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7041-cccc51" Nov 04 05:01:02.677067 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:02.677015 4395 scope.go:122] "RemoveContainer" containerID="e141e13bea2e6707f15ea89922486f5fc61baa12d736073f7a03a3faa53f4804" Nov 04 05:01:02.735193 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:02.735174 4395 scope.go:122] "RemoveContainer" containerID="e141e13bea2e6707f15ea89922486f5fc61baa12d736073f7a03a3faa53f4804" Nov 04 05:01:02.735563 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:01:02.735538 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e141e13bea2e6707f15ea89922486f5fc61baa12d736073f7a03a3faa53f4804\": container with ID starting with e141e13bea2e6707f15ea89922486f5fc61baa12d736073f7a03a3faa53f4804 not found: ID does not exist" containerID="e141e13bea2e6707f15ea89922486f5fc61baa12d736073f7a03a3faa53f4804" Nov 04 05:01:02.735660 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:02.735575 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e141e13bea2e6707f15ea89922486f5fc61baa12d736073f7a03a3faa53f4804"} err="failed to get container status \"e141e13bea2e6707f15ea89922486f5fc61baa12d736073f7a03a3faa53f4804\": rpc error: code = NotFound desc = could not find container \"e141e13bea2e6707f15ea89922486f5fc61baa12d736073f7a03a3faa53f4804\": container with ID starting with e141e13bea2e6707f15ea89922486f5fc61baa12d736073f7a03a3faa53f4804 not found: ID does not exist" Nov 04 05:01:02.928309 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:02.928243 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7053-ecea19" Nov 04 05:01:03.111926 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:03.111902 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d45770b-8c39-4239-b799-32010d6df85b-kube-api-access-hs6lt\" (UniqueName: \"kubernetes.io/projected/7d45770b-8c39-4239-b799-32010d6df85b-kube-api-access-hs6lt\") pod \"7d45770b-8c39-4239-b799-32010d6df85b\" (UID: \"7d45770b-8c39-4239-b799-32010d6df85b\") " Nov 04 05:01:03.113783 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:03.113758 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d45770b-8c39-4239-b799-32010d6df85b-kube-api-access-hs6lt" pod "7d45770b-8c39-4239-b799-32010d6df85b" (UID: "7d45770b-8c39-4239-b799-32010d6df85b"). InnerVolumeSpecName "kube-api-access-hs6lt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:01:03.212892 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:03.212831 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs6lt\" (UniqueName: \"kubernetes.io/projected/7d45770b-8c39-4239-b799-32010d6df85b-kube-api-access-hs6lt\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:03.458798 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:01:03.458767 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 05:01:03.684526 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:03.684482 4395 generic.go:356] "Generic (PLEG): container finished" podID="7d45770b-8c39-4239-b799-32010d6df85b" containerID="98d2ea48bc1660190bba9cfd7892574939948529ad271c5493568987598e4be7" exitCode=137 Nov 04 05:01:03.684685 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:03.684527 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7053-ecea19" Nov 04 05:01:03.684685 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:03.684581 4395 scope.go:122] "RemoveContainer" containerID="98d2ea48bc1660190bba9cfd7892574939948529ad271c5493568987598e4be7" Nov 04 05:01:03.694172 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:03.694155 4395 scope.go:122] "RemoveContainer" containerID="98d2ea48bc1660190bba9cfd7892574939948529ad271c5493568987598e4be7" Nov 04 05:01:03.694474 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:01:03.694456 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98d2ea48bc1660190bba9cfd7892574939948529ad271c5493568987598e4be7\": container with ID starting with 98d2ea48bc1660190bba9cfd7892574939948529ad271c5493568987598e4be7 not found: ID does not exist" containerID="98d2ea48bc1660190bba9cfd7892574939948529ad271c5493568987598e4be7" Nov 04 05:01:03.694560 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:03.694484 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98d2ea48bc1660190bba9cfd7892574939948529ad271c5493568987598e4be7"} err="failed to get container status \"98d2ea48bc1660190bba9cfd7892574939948529ad271c5493568987598e4be7\": rpc error: code = NotFound desc = could not find container \"98d2ea48bc1660190bba9cfd7892574939948529ad271c5493568987598e4be7\": container with ID starting with 98d2ea48bc1660190bba9cfd7892574939948529ad271c5493568987598e4be7 not found: ID does not exist" Nov 04 05:01:03.741271 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:01:03.741245 4395 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7053-ecea19\" is forbidden: User \"system:node:ip-10-0-134-224.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-224.us-west-2.compute.internal' and this object" podUID="7d45770b-8c39-4239-b799-32010d6df85b" pod="opendatahub/test-curl-7053-ecea19" Nov 04 05:01:04.417124 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:04.417090 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d45770b-8c39-4239-b799-32010d6df85b" path="/var/lib/kubelet/pods/7d45770b-8c39-4239-b799-32010d6df85b/volumes" Nov 04 05:01:05.810186 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:05.810155 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-19bf5b/premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b"] Nov 04 05:01:05.856269 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:05.856243 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-19bf5b/simulator-w5-19bf5b-kserve-58fb87c755-dqlhm"] Nov 04 05:01:09.929675 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:09.929637 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-a48f7e/distinct-w4-a48f7e-kserve-8d66794c4-94568" Nov 04 05:01:09.947908 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:09.947875 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-a48f7e/distinct-w4-a48f7e-kserve-8d66794c4-94568" Nov 04 05:01:10.318174 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:10.318137 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-204fa0/premium-w0-204fa0-kserve-6848b499d9-cczvl"] Nov 04 05:01:10.323987 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:10.323959 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-204fa0/simulator-w0-204fa0-kserve-64b5544888-df4pp"] Nov 04 05:01:10.743966 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:10.743883 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a48f7e/distinct-w4-a48f7e-kserve-8d66794c4-94568" Nov 04 05:01:18.850797 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:18.850767 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09028ce6-ktmhx"] Nov 04 05:01:18.851210 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:18.851196 4395 cpu_manager.go:414] "containerMap: removing container" podUID="7d45770b-8c39-4239-b799-32010d6df85b" containerName="test-curl-7053-ecea19" Nov 04 05:01:18.851254 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:18.851211 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d45770b-8c39-4239-b799-32010d6df85b" containerName="test-curl-7053-ecea19" Nov 04 05:01:18.851254 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:18.851238 4395 cpu_manager.go:414] "containerMap: removing container" podUID="b0a64840-53b4-456b-8465-b256b64bfe35" containerName="test-curl-7041-cccc51" Nov 04 05:01:18.851254 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:18.851243 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0a64840-53b4-456b-8465-b256b64bfe35" containerName="test-curl-7041-cccc51" Nov 04 05:01:18.851388 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:18.851345 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0a64840-53b4-456b-8465-b256b64bfe35" containerName="test-curl-7041-cccc51" Nov 04 05:01:18.851388 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:18.851359 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d45770b-8c39-4239-b799-32010d6df85b" containerName="test-curl-7053-ecea19" Nov 04 05:01:18.854594 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:18.854576 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09028ce6-ktmhx" Nov 04 05:01:18.903108 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:18.903088 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 05:01:18.949264 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:18.949239 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nz4vh\"" Nov 04 05:01:18.952054 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:18.952035 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09028ce6-ktmhx"] Nov 04 05:01:18.969821 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:18.969799 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e40da9e-fed2-40ab-b430-9e69b8bab9d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09028ce6-ktmhx\" (UID: \"8e40da9e-fed2-40ab-b430-9e69b8bab9d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09028ce6-ktmhx" Nov 04 05:01:18.969970 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:18.969950 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wvq5\" (UniqueName: \"kubernetes.io/projected/8e40da9e-fed2-40ab-b430-9e69b8bab9d0-kube-api-access-4wvq5\") pod \"maas-api-revoke-sub-09028ce6-ktmhx\" (UID: \"8e40da9e-fed2-40ab-b430-9e69b8bab9d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09028ce6-ktmhx" Nov 04 05:01:18.970017 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:18.969993 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e40da9e-fed2-40ab-b430-9e69b8bab9d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09028ce6-ktmhx\" (UID: \"8e40da9e-fed2-40ab-b430-9e69b8bab9d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09028ce6-ktmhx" Nov 04 05:01:19.070816 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:19.070790 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wvq5\" (UniqueName: \"kubernetes.io/projected/8e40da9e-fed2-40ab-b430-9e69b8bab9d0-kube-api-access-4wvq5\") pod \"maas-api-revoke-sub-09028ce6-ktmhx\" (UID: \"8e40da9e-fed2-40ab-b430-9e69b8bab9d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09028ce6-ktmhx" Nov 04 05:01:19.070916 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:19.070828 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e40da9e-fed2-40ab-b430-9e69b8bab9d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09028ce6-ktmhx\" (UID: \"8e40da9e-fed2-40ab-b430-9e69b8bab9d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09028ce6-ktmhx" Nov 04 05:01:19.070916 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:19.070857 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e40da9e-fed2-40ab-b430-9e69b8bab9d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09028ce6-ktmhx\" (UID: \"8e40da9e-fed2-40ab-b430-9e69b8bab9d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09028ce6-ktmhx" Nov 04 05:01:19.071511 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:19.071488 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e40da9e-fed2-40ab-b430-9e69b8bab9d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09028ce6-ktmhx\" (UID: \"8e40da9e-fed2-40ab-b430-9e69b8bab9d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09028ce6-ktmhx" Nov 04 05:01:19.073155 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:19.073134 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e40da9e-fed2-40ab-b430-9e69b8bab9d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09028ce6-ktmhx\" (UID: \"8e40da9e-fed2-40ab-b430-9e69b8bab9d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09028ce6-ktmhx" Nov 04 05:01:19.128026 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:19.127974 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wvq5\" (UniqueName: \"kubernetes.io/projected/8e40da9e-fed2-40ab-b430-9e69b8bab9d0-kube-api-access-4wvq5\") pod \"maas-api-revoke-sub-09028ce6-ktmhx\" (UID: \"8e40da9e-fed2-40ab-b430-9e69b8bab9d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09028ce6-ktmhx" Nov 04 05:01:19.166943 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:19.166919 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09028ce6-ktmhx" Nov 04 05:01:19.547135 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:19.547107 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09028ce6-ktmhx"] Nov 04 05:01:19.548406 ip-10-0-134-224 kubenswrapper[4395]: W1104 05:01:19.548377 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e40da9e_fed2_40ab_b430_9e69b8bab9d0.slice/crio-31b09ea12774472c73c69c995c27afe1a4ff971ed08089ed5c202040228512bb WatchSource:0}: Error finding container 31b09ea12774472c73c69c995c27afe1a4ff971ed08089ed5c202040228512bb: Status 404 returned error can't find the container with id 31b09ea12774472c73c69c995c27afe1a4ff971ed08089ed5c202040228512bb Nov 04 05:01:19.786781 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:19.786746 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09028ce6-ktmhx" event={"ID":"8e40da9e-fed2-40ab-b430-9e69b8bab9d0","Type":"ContainerStarted","Data":"31b09ea12774472c73c69c995c27afe1a4ff971ed08089ed5c202040228512bb"} Nov 04 05:01:20.795951 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:20.795913 4395 generic.go:356] "Generic (PLEG): container finished" podID="8e40da9e-fed2-40ab-b430-9e69b8bab9d0" containerID="aa87a649cdf3fbd9114bf57ca2be66909bbac2bde62592b064eaff317209c9ae" exitCode=0 Nov 04 05:01:20.796480 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:20.795992 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09028ce6-ktmhx" event={"ID":"8e40da9e-fed2-40ab-b430-9e69b8bab9d0","Type":"ContainerDied","Data":"aa87a649cdf3fbd9114bf57ca2be66909bbac2bde62592b064eaff317209c9ae"} Nov 04 05:01:20.830582 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:20.830553 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-19bf5b/premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b" podUID="ac06b52c-cb97-479d-813f-a9e9d6d9e67a" containerName="main" containerID="cri-o://4ec74aec84bce2314467d02a0e4dd56849cc9f7bba750627c753a0e812b8caf6" gracePeriod=45 Nov 04 05:01:20.865852 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:20.865820 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-19bf5b/premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b" podUID="ac06b52c-cb97-479d-813f-a9e9d6d9e67a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": dial tcp 10.134.0.54:8000: connect: connection refused" Nov 04 05:01:20.875453 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:20.875429 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-19bf5b/simulator-w5-19bf5b-kserve-58fb87c755-dqlhm" podUID="e577d1b4-d707-40c7-a78f-c64c480c1af2" containerName="main" containerID="cri-o://857cd29c43cc69cd1b92e425e5bb73cbcbc000e4684fb48719cdebab93601af6" gracePeriod=45 Nov 04 05:01:21.173506 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.173480 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-19bf5b/premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b" Nov 04 05:01:21.176183 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.176163 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-19bf5b/simulator-w5-19bf5b-kserve-58fb87c755-dqlhm" Nov 04 05:01:21.293096 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.292998 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-tmp-dir\") pod \"ac06b52c-cb97-479d-813f-a9e9d6d9e67a\" (UID: \"ac06b52c-cb97-479d-813f-a9e9d6d9e67a\") " Nov 04 05:01:21.293096 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.293060 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-tls-certs\" (UniqueName: \"kubernetes.io/secret/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-tls-certs\") pod \"ac06b52c-cb97-479d-813f-a9e9d6d9e67a\" (UID: \"ac06b52c-cb97-479d-813f-a9e9d6d9e67a\") " Nov 04 05:01:21.293096 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.293105 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e577d1b4-d707-40c7-a78f-c64c480c1af2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e577d1b4-d707-40c7-a78f-c64c480c1af2-model-cache\") pod \"e577d1b4-d707-40c7-a78f-c64c480c1af2\" (UID: \"e577d1b4-d707-40c7-a78f-c64c480c1af2\") " Nov 04 05:01:21.293447 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.293166 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e577d1b4-d707-40c7-a78f-c64c480c1af2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e577d1b4-d707-40c7-a78f-c64c480c1af2-tmp-dir\") pod \"e577d1b4-d707-40c7-a78f-c64c480c1af2\" (UID: \"e577d1b4-d707-40c7-a78f-c64c480c1af2\") " Nov 04 05:01:21.293447 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.293196 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-kube-api-access-zrq64\" (UniqueName: \"kubernetes.io/projected/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-kube-api-access-zrq64\") pod \"ac06b52c-cb97-479d-813f-a9e9d6d9e67a\" (UID: \"ac06b52c-cb97-479d-813f-a9e9d6d9e67a\") " Nov 04 05:01:21.293447 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.293230 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-dshm\") pod \"ac06b52c-cb97-479d-813f-a9e9d6d9e67a\" (UID: \"ac06b52c-cb97-479d-813f-a9e9d6d9e67a\") " Nov 04 05:01:21.293447 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.293280 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-home\" (UniqueName: \"kubernetes.io/empty-dir/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-home\") pod \"ac06b52c-cb97-479d-813f-a9e9d6d9e67a\" (UID: \"ac06b52c-cb97-479d-813f-a9e9d6d9e67a\") " Nov 04 05:01:21.293447 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.293338 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e577d1b4-d707-40c7-a78f-c64c480c1af2-home\" (UniqueName: \"kubernetes.io/empty-dir/e577d1b4-d707-40c7-a78f-c64c480c1af2-home\") pod \"e577d1b4-d707-40c7-a78f-c64c480c1af2\" (UID: \"e577d1b4-d707-40c7-a78f-c64c480c1af2\") " Nov 04 05:01:21.293447 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.293436 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e577d1b4-d707-40c7-a78f-c64c480c1af2-kube-api-access-czlj2\" (UniqueName: \"kubernetes.io/projected/e577d1b4-d707-40c7-a78f-c64c480c1af2-kube-api-access-czlj2\") pod \"e577d1b4-d707-40c7-a78f-c64c480c1af2\" (UID: \"e577d1b4-d707-40c7-a78f-c64c480c1af2\") " Nov 04 05:01:21.293802 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.293479 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-model-cache\") pod \"ac06b52c-cb97-479d-813f-a9e9d6d9e67a\" (UID: \"ac06b52c-cb97-479d-813f-a9e9d6d9e67a\") " Nov 04 05:01:21.293802 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.293530 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e577d1b4-d707-40c7-a78f-c64c480c1af2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e577d1b4-d707-40c7-a78f-c64c480c1af2-dshm\") pod \"e577d1b4-d707-40c7-a78f-c64c480c1af2\" (UID: \"e577d1b4-d707-40c7-a78f-c64c480c1af2\") " Nov 04 05:01:21.293802 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.293568 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e577d1b4-d707-40c7-a78f-c64c480c1af2-tls-certs\" (UniqueName: \"kubernetes.io/secret/e577d1b4-d707-40c7-a78f-c64c480c1af2-tls-certs\") pod \"e577d1b4-d707-40c7-a78f-c64c480c1af2\" (UID: \"e577d1b4-d707-40c7-a78f-c64c480c1af2\") " Nov 04 05:01:21.294742 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.294532 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e577d1b4-d707-40c7-a78f-c64c480c1af2-tmp-dir" pod "e577d1b4-d707-40c7-a78f-c64c480c1af2" (UID: "e577d1b4-d707-40c7-a78f-c64c480c1af2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:01:21.294992 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.294875 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-tmp-dir" pod "ac06b52c-cb97-479d-813f-a9e9d6d9e67a" (UID: "ac06b52c-cb97-479d-813f-a9e9d6d9e67a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:01:21.294992 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.294922 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-home" pod "ac06b52c-cb97-479d-813f-a9e9d6d9e67a" (UID: "ac06b52c-cb97-479d-813f-a9e9d6d9e67a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:01:21.295350 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.295283 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e577d1b4-d707-40c7-a78f-c64c480c1af2-model-cache" pod "e577d1b4-d707-40c7-a78f-c64c480c1af2" (UID: "e577d1b4-d707-40c7-a78f-c64c480c1af2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:01:21.295350 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.295285 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-model-cache" pod "ac06b52c-cb97-479d-813f-a9e9d6d9e67a" (UID: "ac06b52c-cb97-479d-813f-a9e9d6d9e67a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:01:21.300334 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.297374 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e577d1b4-d707-40c7-a78f-c64c480c1af2-home" pod "e577d1b4-d707-40c7-a78f-c64c480c1af2" (UID: "e577d1b4-d707-40c7-a78f-c64c480c1af2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:01:21.300334 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.298667 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-kube-api-access-zrq64" pod "ac06b52c-cb97-479d-813f-a9e9d6d9e67a" (UID: "ac06b52c-cb97-479d-813f-a9e9d6d9e67a"). InnerVolumeSpecName "kube-api-access-zrq64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:01:21.300334 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.298935 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e577d1b4-d707-40c7-a78f-c64c480c1af2-tls-certs" pod "e577d1b4-d707-40c7-a78f-c64c480c1af2" (UID: "e577d1b4-d707-40c7-a78f-c64c480c1af2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:01:21.301900 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.301861 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-tls-certs" pod "ac06b52c-cb97-479d-813f-a9e9d6d9e67a" (UID: "ac06b52c-cb97-479d-813f-a9e9d6d9e67a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:01:21.302004 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.301910 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e577d1b4-d707-40c7-a78f-c64c480c1af2-dshm" pod "e577d1b4-d707-40c7-a78f-c64c480c1af2" (UID: "e577d1b4-d707-40c7-a78f-c64c480c1af2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:01:21.302004 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.301912 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e577d1b4-d707-40c7-a78f-c64c480c1af2-kube-api-access-czlj2" pod "e577d1b4-d707-40c7-a78f-c64c480c1af2" (UID: "e577d1b4-d707-40c7-a78f-c64c480c1af2"). InnerVolumeSpecName "kube-api-access-czlj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:01:21.304472 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.304446 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-dshm" pod "ac06b52c-cb97-479d-813f-a9e9d6d9e67a" (UID: "ac06b52c-cb97-479d-813f-a9e9d6d9e67a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:01:21.394792 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.394704 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czlj2\" (UniqueName: \"kubernetes.io/projected/e577d1b4-d707-40c7-a78f-c64c480c1af2-kube-api-access-czlj2\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:21.394792 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.394732 4395 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-model-cache\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:21.394792 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.394742 4395 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e577d1b4-d707-40c7-a78f-c64c480c1af2-dshm\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:21.394792 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.394751 4395 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e577d1b4-d707-40c7-a78f-c64c480c1af2-tls-certs\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:21.394792 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.394759 4395 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-tmp-dir\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:21.394792 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.394767 4395 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-tls-certs\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:21.394792 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.394775 4395 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e577d1b4-d707-40c7-a78f-c64c480c1af2-model-cache\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:21.394792 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.394783 4395 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e577d1b4-d707-40c7-a78f-c64c480c1af2-tmp-dir\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:21.394792 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.394791 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrq64\" (UniqueName: \"kubernetes.io/projected/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-kube-api-access-zrq64\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:21.394792 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.394800 4395 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-dshm\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:21.395407 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.394809 4395 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac06b52c-cb97-479d-813f-a9e9d6d9e67a-home\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:21.395407 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.394816 4395 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e577d1b4-d707-40c7-a78f-c64c480c1af2-home\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:21.805140 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.805100 4395 generic.go:356] "Generic (PLEG): container finished" podID="e577d1b4-d707-40c7-a78f-c64c480c1af2" containerID="857cd29c43cc69cd1b92e425e5bb73cbcbc000e4684fb48719cdebab93601af6" exitCode=0 Nov 04 05:01:21.805602 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.805151 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-19bf5b/simulator-w5-19bf5b-kserve-58fb87c755-dqlhm" Nov 04 05:01:21.805602 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.805182 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-19bf5b/simulator-w5-19bf5b-kserve-58fb87c755-dqlhm" event={"ID":"e577d1b4-d707-40c7-a78f-c64c480c1af2","Type":"ContainerDied","Data":"857cd29c43cc69cd1b92e425e5bb73cbcbc000e4684fb48719cdebab93601af6"} Nov 04 05:01:21.805602 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.805227 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-19bf5b/simulator-w5-19bf5b-kserve-58fb87c755-dqlhm" event={"ID":"e577d1b4-d707-40c7-a78f-c64c480c1af2","Type":"ContainerDied","Data":"38fdf2301d66b04b7508fab27336c7dd0720564755c6882bcd21993c1a027913"} Nov 04 05:01:21.805602 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.805247 4395 scope.go:122] "RemoveContainer" containerID="857cd29c43cc69cd1b92e425e5bb73cbcbc000e4684fb48719cdebab93601af6" Nov 04 05:01:21.806807 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.806782 4395 generic.go:356] "Generic (PLEG): container finished" podID="ac06b52c-cb97-479d-813f-a9e9d6d9e67a" containerID="4ec74aec84bce2314467d02a0e4dd56849cc9f7bba750627c753a0e812b8caf6" exitCode=0 Nov 04 05:01:21.806937 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.806826 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-19bf5b/premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b" Nov 04 05:01:21.806937 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.806868 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-19bf5b/premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b" event={"ID":"ac06b52c-cb97-479d-813f-a9e9d6d9e67a","Type":"ContainerDied","Data":"4ec74aec84bce2314467d02a0e4dd56849cc9f7bba750627c753a0e812b8caf6"} Nov 04 05:01:21.806937 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.806897 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-19bf5b/premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b" event={"ID":"ac06b52c-cb97-479d-813f-a9e9d6d9e67a","Type":"ContainerDied","Data":"5f2e269cbf7447b81b48a184f3a079f74199c8de2a34b0a6a79bc7e4ec48006c"} Nov 04 05:01:21.815984 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.815967 4395 scope.go:122] "RemoveContainer" containerID="857cd29c43cc69cd1b92e425e5bb73cbcbc000e4684fb48719cdebab93601af6" Nov 04 05:01:21.816278 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:01:21.816257 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"857cd29c43cc69cd1b92e425e5bb73cbcbc000e4684fb48719cdebab93601af6\": container with ID starting with 857cd29c43cc69cd1b92e425e5bb73cbcbc000e4684fb48719cdebab93601af6 not found: ID does not exist" containerID="857cd29c43cc69cd1b92e425e5bb73cbcbc000e4684fb48719cdebab93601af6" Nov 04 05:01:21.816372 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.816281 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"857cd29c43cc69cd1b92e425e5bb73cbcbc000e4684fb48719cdebab93601af6"} err="failed to get container status \"857cd29c43cc69cd1b92e425e5bb73cbcbc000e4684fb48719cdebab93601af6\": rpc error: code = NotFound desc = could not find container \"857cd29c43cc69cd1b92e425e5bb73cbcbc000e4684fb48719cdebab93601af6\": container with ID starting with 857cd29c43cc69cd1b92e425e5bb73cbcbc000e4684fb48719cdebab93601af6 not found: ID does not exist" Nov 04 05:01:21.816372 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.816310 4395 scope.go:122] "RemoveContainer" containerID="4ec74aec84bce2314467d02a0e4dd56849cc9f7bba750627c753a0e812b8caf6" Nov 04 05:01:21.824850 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.824832 4395 scope.go:122] "RemoveContainer" containerID="4ec74aec84bce2314467d02a0e4dd56849cc9f7bba750627c753a0e812b8caf6" Nov 04 05:01:21.825137 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:01:21.825118 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ec74aec84bce2314467d02a0e4dd56849cc9f7bba750627c753a0e812b8caf6\": container with ID starting with 4ec74aec84bce2314467d02a0e4dd56849cc9f7bba750627c753a0e812b8caf6 not found: ID does not exist" containerID="4ec74aec84bce2314467d02a0e4dd56849cc9f7bba750627c753a0e812b8caf6" Nov 04 05:01:21.825207 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.825142 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec74aec84bce2314467d02a0e4dd56849cc9f7bba750627c753a0e812b8caf6"} err="failed to get container status \"4ec74aec84bce2314467d02a0e4dd56849cc9f7bba750627c753a0e812b8caf6\": rpc error: code = NotFound desc = could not find container \"4ec74aec84bce2314467d02a0e4dd56849cc9f7bba750627c753a0e812b8caf6\": container with ID starting with 4ec74aec84bce2314467d02a0e4dd56849cc9f7bba750627c753a0e812b8caf6 not found: ID does not exist" Nov 04 05:01:21.944322 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.944286 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09028ce6-ktmhx" Nov 04 05:01:21.969858 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.969829 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-19bf5b/simulator-w5-19bf5b-kserve-58fb87c755-dqlhm"] Nov 04 05:01:21.972770 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:21.972745 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-19bf5b/simulator-w5-19bf5b-kserve-58fb87c755-dqlhm"] Nov 04 05:01:22.020071 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:22.020044 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e40da9e-fed2-40ab-b430-9e69b8bab9d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e40da9e-fed2-40ab-b430-9e69b8bab9d0-maas-api-openshift-ca\") pod \"8e40da9e-fed2-40ab-b430-9e69b8bab9d0\" (UID: \"8e40da9e-fed2-40ab-b430-9e69b8bab9d0\") " Nov 04 05:01:22.020316 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:22.020268 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e40da9e-fed2-40ab-b430-9e69b8bab9d0-kube-api-access-4wvq5\" (UniqueName: \"kubernetes.io/projected/8e40da9e-fed2-40ab-b430-9e69b8bab9d0-kube-api-access-4wvq5\") pod \"8e40da9e-fed2-40ab-b430-9e69b8bab9d0\" (UID: \"8e40da9e-fed2-40ab-b430-9e69b8bab9d0\") " Nov 04 05:01:22.020456 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:22.020433 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e40da9e-fed2-40ab-b430-9e69b8bab9d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e40da9e-fed2-40ab-b430-9e69b8bab9d0-maas-api-xks-ca\") pod \"8e40da9e-fed2-40ab-b430-9e69b8bab9d0\" (UID: \"8e40da9e-fed2-40ab-b430-9e69b8bab9d0\") " Nov 04 05:01:22.020608 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:22.020558 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e40da9e-fed2-40ab-b430-9e69b8bab9d0-maas-api-openshift-ca" pod "8e40da9e-fed2-40ab-b430-9e69b8bab9d0" (UID: "8e40da9e-fed2-40ab-b430-9e69b8bab9d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:01:22.021476 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:22.021451 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e40da9e-fed2-40ab-b430-9e69b8bab9d0-maas-api-openshift-ca\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:22.022551 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:22.022531 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e40da9e-fed2-40ab-b430-9e69b8bab9d0-kube-api-access-4wvq5" pod "8e40da9e-fed2-40ab-b430-9e69b8bab9d0" (UID: "8e40da9e-fed2-40ab-b430-9e69b8bab9d0"). InnerVolumeSpecName "kube-api-access-4wvq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:01:22.022695 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:22.022677 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e40da9e-fed2-40ab-b430-9e69b8bab9d0-maas-api-xks-ca" pod "8e40da9e-fed2-40ab-b430-9e69b8bab9d0" (UID: "8e40da9e-fed2-40ab-b430-9e69b8bab9d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:01:22.079162 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:22.079086 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-19bf5b/premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b"] Nov 04 05:01:22.082704 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:22.082677 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-19bf5b/premium-w5-19bf5b-kserve-7f8cdbb86b-nn28b"] Nov 04 05:01:22.122561 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:22.122537 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wvq5\" (UniqueName: \"kubernetes.io/projected/8e40da9e-fed2-40ab-b430-9e69b8bab9d0-kube-api-access-4wvq5\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:22.122561 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:22.122560 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e40da9e-fed2-40ab-b430-9e69b8bab9d0-maas-api-xks-ca\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:22.416487 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:22.416402 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac06b52c-cb97-479d-813f-a9e9d6d9e67a" path="/var/lib/kubelet/pods/ac06b52c-cb97-479d-813f-a9e9d6d9e67a/volumes" Nov 04 05:01:22.416919 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:22.416896 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e577d1b4-d707-40c7-a78f-c64c480c1af2" path="/var/lib/kubelet/pods/e577d1b4-d707-40c7-a78f-c64c480c1af2/volumes" Nov 04 05:01:22.815340 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:22.815312 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09028ce6-ktmhx" Nov 04 05:01:22.815818 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:22.815341 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09028ce6-ktmhx" event={"ID":"8e40da9e-fed2-40ab-b430-9e69b8bab9d0","Type":"ContainerDied","Data":"31b09ea12774472c73c69c995c27afe1a4ff971ed08089ed5c202040228512bb"} Nov 04 05:01:22.815818 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:22.815401 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31b09ea12774472c73c69c995c27afe1a4ff971ed08089ed5c202040228512bb" Nov 04 05:01:22.945276 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:22.945250 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-db8fae/premium-w6-db8fae-kserve-6c464dbc8b-xvnsz"] Nov 04 05:01:24.012503 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:24.012466 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-19bf5b-7c6d784c4-s8rpn"] Nov 04 05:01:25.339259 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.339216 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-204fa0/premium-w0-204fa0-kserve-6848b499d9-cczvl" podUID="37ed0b35-a2e9-49eb-95c5-793da9836c6f" containerName="main" containerID="cri-o://85452ff402f1088b830a0533632656779de6e5e054f427f4ca13b53e43e624fc" gracePeriod=45 Nov 04 05:01:25.343231 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.343201 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-204fa0/simulator-w0-204fa0-kserve-64b5544888-df4pp" podUID="80a73743-0113-4f1c-94cc-38b67249b698" containerName="main" containerID="cri-o://7bc11ae9ec0cd0b2bcc5789c5c3470f5c1d11553ec6fdb3fc1f9d80017373fbc" gracePeriod=45 Nov 04 05:01:25.431929 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.431891 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-204fa0/simulator-w0-204fa0-kserve-64b5544888-df4pp" podUID="80a73743-0113-4f1c-94cc-38b67249b698" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": dial tcp 10.134.0.53:8000: connect: connection refused" Nov 04 05:01:25.636955 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.636934 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-204fa0/premium-w0-204fa0-kserve-6848b499d9-cczvl" Nov 04 05:01:25.694821 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.694798 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-204fa0/simulator-w0-204fa0-kserve-64b5544888-df4pp" Nov 04 05:01:25.790751 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.790696 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37ed0b35-a2e9-49eb-95c5-793da9836c6f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37ed0b35-a2e9-49eb-95c5-793da9836c6f-tmp-dir\") pod \"37ed0b35-a2e9-49eb-95c5-793da9836c6f\" (UID: \"37ed0b35-a2e9-49eb-95c5-793da9836c6f\") " Nov 04 05:01:25.790950 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.790786 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37ed0b35-a2e9-49eb-95c5-793da9836c6f-home\" (UniqueName: \"kubernetes.io/empty-dir/37ed0b35-a2e9-49eb-95c5-793da9836c6f-home\") pod \"37ed0b35-a2e9-49eb-95c5-793da9836c6f\" (UID: \"37ed0b35-a2e9-49eb-95c5-793da9836c6f\") " Nov 04 05:01:25.790950 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.790860 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80a73743-0113-4f1c-94cc-38b67249b698-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80a73743-0113-4f1c-94cc-38b67249b698-model-cache\") pod \"80a73743-0113-4f1c-94cc-38b67249b698\" (UID: \"80a73743-0113-4f1c-94cc-38b67249b698\") " Nov 04 05:01:25.790950 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.790899 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80a73743-0113-4f1c-94cc-38b67249b698-kube-api-access-tfts2\" (UniqueName: \"kubernetes.io/projected/80a73743-0113-4f1c-94cc-38b67249b698-kube-api-access-tfts2\") pod \"80a73743-0113-4f1c-94cc-38b67249b698\" (UID: \"80a73743-0113-4f1c-94cc-38b67249b698\") " Nov 04 05:01:25.790950 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.790930 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80a73743-0113-4f1c-94cc-38b67249b698-tls-certs\" (UniqueName: \"kubernetes.io/secret/80a73743-0113-4f1c-94cc-38b67249b698-tls-certs\") pod \"80a73743-0113-4f1c-94cc-38b67249b698\" (UID: \"80a73743-0113-4f1c-94cc-38b67249b698\") " Nov 04 05:01:25.791199 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.790996 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80a73743-0113-4f1c-94cc-38b67249b698-dshm\" (UniqueName: \"kubernetes.io/empty-dir/80a73743-0113-4f1c-94cc-38b67249b698-dshm\") pod \"80a73743-0113-4f1c-94cc-38b67249b698\" (UID: \"80a73743-0113-4f1c-94cc-38b67249b698\") " Nov 04 05:01:25.791199 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.791026 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37ed0b35-a2e9-49eb-95c5-793da9836c6f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37ed0b35-a2e9-49eb-95c5-793da9836c6f-model-cache\") pod \"37ed0b35-a2e9-49eb-95c5-793da9836c6f\" (UID: \"37ed0b35-a2e9-49eb-95c5-793da9836c6f\") " Nov 04 05:01:25.791199 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.791044 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ed0b35-a2e9-49eb-95c5-793da9836c6f-tmp-dir" pod "37ed0b35-a2e9-49eb-95c5-793da9836c6f" (UID: "37ed0b35-a2e9-49eb-95c5-793da9836c6f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:01:25.791199 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.791069 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37ed0b35-a2e9-49eb-95c5-793da9836c6f-kube-api-access-ngcg5\" (UniqueName: \"kubernetes.io/projected/37ed0b35-a2e9-49eb-95c5-793da9836c6f-kube-api-access-ngcg5\") pod \"37ed0b35-a2e9-49eb-95c5-793da9836c6f\" (UID: \"37ed0b35-a2e9-49eb-95c5-793da9836c6f\") " Nov 04 05:01:25.791199 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.791176 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80a73743-0113-4f1c-94cc-38b67249b698-home\" (UniqueName: \"kubernetes.io/empty-dir/80a73743-0113-4f1c-94cc-38b67249b698-home\") pod \"80a73743-0113-4f1c-94cc-38b67249b698\" (UID: \"80a73743-0113-4f1c-94cc-38b67249b698\") " Nov 04 05:01:25.791472 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.791211 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37ed0b35-a2e9-49eb-95c5-793da9836c6f-tls-certs\" (UniqueName: \"kubernetes.io/secret/37ed0b35-a2e9-49eb-95c5-793da9836c6f-tls-certs\") pod \"37ed0b35-a2e9-49eb-95c5-793da9836c6f\" (UID: \"37ed0b35-a2e9-49eb-95c5-793da9836c6f\") " Nov 04 05:01:25.791472 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.791253 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37ed0b35-a2e9-49eb-95c5-793da9836c6f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/37ed0b35-a2e9-49eb-95c5-793da9836c6f-dshm\") pod \"37ed0b35-a2e9-49eb-95c5-793da9836c6f\" (UID: \"37ed0b35-a2e9-49eb-95c5-793da9836c6f\") " Nov 04 05:01:25.791472 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.791260 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ed0b35-a2e9-49eb-95c5-793da9836c6f-model-cache" pod "37ed0b35-a2e9-49eb-95c5-793da9836c6f" (UID: "37ed0b35-a2e9-49eb-95c5-793da9836c6f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:01:25.791472 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.791310 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80a73743-0113-4f1c-94cc-38b67249b698-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80a73743-0113-4f1c-94cc-38b67249b698-tmp-dir\") pod \"80a73743-0113-4f1c-94cc-38b67249b698\" (UID: \"80a73743-0113-4f1c-94cc-38b67249b698\") " Nov 04 05:01:25.791472 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.791420 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ed0b35-a2e9-49eb-95c5-793da9836c6f-home" pod "37ed0b35-a2e9-49eb-95c5-793da9836c6f" (UID: "37ed0b35-a2e9-49eb-95c5-793da9836c6f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:01:25.791726 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.791471 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80a73743-0113-4f1c-94cc-38b67249b698-home" pod "80a73743-0113-4f1c-94cc-38b67249b698" (UID: "80a73743-0113-4f1c-94cc-38b67249b698"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:01:25.791726 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.791688 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80a73743-0113-4f1c-94cc-38b67249b698-model-cache" pod "80a73743-0113-4f1c-94cc-38b67249b698" (UID: "80a73743-0113-4f1c-94cc-38b67249b698"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:01:25.791840 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.791779 4395 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37ed0b35-a2e9-49eb-95c5-793da9836c6f-tmp-dir\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:25.791840 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.791801 4395 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37ed0b35-a2e9-49eb-95c5-793da9836c6f-home\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:25.791840 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.791817 4395 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80a73743-0113-4f1c-94cc-38b67249b698-model-cache\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:25.791840 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.791832 4395 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37ed0b35-a2e9-49eb-95c5-793da9836c6f-model-cache\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:25.792052 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.791847 4395 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80a73743-0113-4f1c-94cc-38b67249b698-home\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:25.792109 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.792055 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80a73743-0113-4f1c-94cc-38b67249b698-tmp-dir" pod "80a73743-0113-4f1c-94cc-38b67249b698" (UID: "80a73743-0113-4f1c-94cc-38b67249b698"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:01:25.794430 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.794397 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80a73743-0113-4f1c-94cc-38b67249b698-tls-certs" pod "80a73743-0113-4f1c-94cc-38b67249b698" (UID: "80a73743-0113-4f1c-94cc-38b67249b698"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:01:25.799645 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.799615 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37ed0b35-a2e9-49eb-95c5-793da9836c6f-tls-certs" pod "37ed0b35-a2e9-49eb-95c5-793da9836c6f" (UID: "37ed0b35-a2e9-49eb-95c5-793da9836c6f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:01:25.799645 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.799630 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37ed0b35-a2e9-49eb-95c5-793da9836c6f-kube-api-access-ngcg5" pod "37ed0b35-a2e9-49eb-95c5-793da9836c6f" (UID: "37ed0b35-a2e9-49eb-95c5-793da9836c6f"). InnerVolumeSpecName "kube-api-access-ngcg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:01:25.799645 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.799636 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80a73743-0113-4f1c-94cc-38b67249b698-dshm" pod "80a73743-0113-4f1c-94cc-38b67249b698" (UID: "80a73743-0113-4f1c-94cc-38b67249b698"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:01:25.799802 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.799659 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80a73743-0113-4f1c-94cc-38b67249b698-kube-api-access-tfts2" pod "80a73743-0113-4f1c-94cc-38b67249b698" (UID: "80a73743-0113-4f1c-94cc-38b67249b698"). InnerVolumeSpecName "kube-api-access-tfts2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:01:25.800743 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.800725 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ed0b35-a2e9-49eb-95c5-793da9836c6f-dshm" pod "37ed0b35-a2e9-49eb-95c5-793da9836c6f" (UID: "37ed0b35-a2e9-49eb-95c5-793da9836c6f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:01:25.836020 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.835994 4395 generic.go:356] "Generic (PLEG): container finished" podID="37ed0b35-a2e9-49eb-95c5-793da9836c6f" containerID="85452ff402f1088b830a0533632656779de6e5e054f427f4ca13b53e43e624fc" exitCode=0 Nov 04 05:01:25.836278 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.836053 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-204fa0/premium-w0-204fa0-kserve-6848b499d9-cczvl" Nov 04 05:01:25.836278 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.836078 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-204fa0/premium-w0-204fa0-kserve-6848b499d9-cczvl" event={"ID":"37ed0b35-a2e9-49eb-95c5-793da9836c6f","Type":"ContainerDied","Data":"85452ff402f1088b830a0533632656779de6e5e054f427f4ca13b53e43e624fc"} Nov 04 05:01:25.836278 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.836128 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-204fa0/premium-w0-204fa0-kserve-6848b499d9-cczvl" event={"ID":"37ed0b35-a2e9-49eb-95c5-793da9836c6f","Type":"ContainerDied","Data":"2b5ae8f393aa5ba69b0e9a87794756499a10aed835f96ce1d5929f803dd8b308"} Nov 04 05:01:25.836278 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.836156 4395 scope.go:122] "RemoveContainer" containerID="85452ff402f1088b830a0533632656779de6e5e054f427f4ca13b53e43e624fc" Nov 04 05:01:25.837892 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.837868 4395 generic.go:356] "Generic (PLEG): container finished" podID="80a73743-0113-4f1c-94cc-38b67249b698" containerID="7bc11ae9ec0cd0b2bcc5789c5c3470f5c1d11553ec6fdb3fc1f9d80017373fbc" exitCode=0 Nov 04 05:01:25.837984 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.837904 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-204fa0/simulator-w0-204fa0-kserve-64b5544888-df4pp" event={"ID":"80a73743-0113-4f1c-94cc-38b67249b698","Type":"ContainerDied","Data":"7bc11ae9ec0cd0b2bcc5789c5c3470f5c1d11553ec6fdb3fc1f9d80017373fbc"} Nov 04 05:01:25.837984 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.837920 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-204fa0/simulator-w0-204fa0-kserve-64b5544888-df4pp" Nov 04 05:01:25.837984 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.837942 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-204fa0/simulator-w0-204fa0-kserve-64b5544888-df4pp" event={"ID":"80a73743-0113-4f1c-94cc-38b67249b698","Type":"ContainerDied","Data":"48c812fb6f66537c00cd572c18202f5c8cb8c842b2a0437c08218289c1a117d9"} Nov 04 05:01:25.852682 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.852657 4395 scope.go:122] "RemoveContainer" containerID="85452ff402f1088b830a0533632656779de6e5e054f427f4ca13b53e43e624fc" Nov 04 05:01:25.853001 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:01:25.852981 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85452ff402f1088b830a0533632656779de6e5e054f427f4ca13b53e43e624fc\": container with ID starting with 85452ff402f1088b830a0533632656779de6e5e054f427f4ca13b53e43e624fc not found: ID does not exist" containerID="85452ff402f1088b830a0533632656779de6e5e054f427f4ca13b53e43e624fc" Nov 04 05:01:25.853082 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.853011 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85452ff402f1088b830a0533632656779de6e5e054f427f4ca13b53e43e624fc"} err="failed to get container status \"85452ff402f1088b830a0533632656779de6e5e054f427f4ca13b53e43e624fc\": rpc error: code = NotFound desc = could not find container \"85452ff402f1088b830a0533632656779de6e5e054f427f4ca13b53e43e624fc\": container with ID starting with 85452ff402f1088b830a0533632656779de6e5e054f427f4ca13b53e43e624fc not found: ID does not exist" Nov 04 05:01:25.853082 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.853034 4395 scope.go:122] "RemoveContainer" containerID="7bc11ae9ec0cd0b2bcc5789c5c3470f5c1d11553ec6fdb3fc1f9d80017373fbc" Nov 04 05:01:25.865813 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.865783 4395 scope.go:122] "RemoveContainer" containerID="7bc11ae9ec0cd0b2bcc5789c5c3470f5c1d11553ec6fdb3fc1f9d80017373fbc" Nov 04 05:01:25.866160 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:01:25.866140 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bc11ae9ec0cd0b2bcc5789c5c3470f5c1d11553ec6fdb3fc1f9d80017373fbc\": container with ID starting with 7bc11ae9ec0cd0b2bcc5789c5c3470f5c1d11553ec6fdb3fc1f9d80017373fbc not found: ID does not exist" containerID="7bc11ae9ec0cd0b2bcc5789c5c3470f5c1d11553ec6fdb3fc1f9d80017373fbc" Nov 04 05:01:25.866250 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.866165 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bc11ae9ec0cd0b2bcc5789c5c3470f5c1d11553ec6fdb3fc1f9d80017373fbc"} err="failed to get container status \"7bc11ae9ec0cd0b2bcc5789c5c3470f5c1d11553ec6fdb3fc1f9d80017373fbc\": rpc error: code = NotFound desc = could not find container \"7bc11ae9ec0cd0b2bcc5789c5c3470f5c1d11553ec6fdb3fc1f9d80017373fbc\": container with ID starting with 7bc11ae9ec0cd0b2bcc5789c5c3470f5c1d11553ec6fdb3fc1f9d80017373fbc not found: ID does not exist" Nov 04 05:01:25.893700 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.893601 4395 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80a73743-0113-4f1c-94cc-38b67249b698-dshm\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:25.893947 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.893907 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngcg5\" (UniqueName: \"kubernetes.io/projected/37ed0b35-a2e9-49eb-95c5-793da9836c6f-kube-api-access-ngcg5\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:25.894113 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.894081 4395 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37ed0b35-a2e9-49eb-95c5-793da9836c6f-tls-certs\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:25.894224 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.894211 4395 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37ed0b35-a2e9-49eb-95c5-793da9836c6f-dshm\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:25.894404 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.894375 4395 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80a73743-0113-4f1c-94cc-38b67249b698-tmp-dir\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:25.894511 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.894498 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfts2\" (UniqueName: \"kubernetes.io/projected/80a73743-0113-4f1c-94cc-38b67249b698-kube-api-access-tfts2\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:25.894610 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.894597 4395 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80a73743-0113-4f1c-94cc-38b67249b698-tls-certs\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:25.962724 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.962694 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-204fa0/simulator-w0-204fa0-kserve-64b5544888-df4pp"] Nov 04 05:01:25.966854 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:25.966821 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-204fa0/simulator-w0-204fa0-kserve-64b5544888-df4pp"] Nov 04 05:01:26.071979 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:26.071945 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-204fa0/premium-w0-204fa0-kserve-6848b499d9-cczvl"] Nov 04 05:01:26.077236 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:26.077209 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-204fa0/premium-w0-204fa0-kserve-6848b499d9-cczvl"] Nov 04 05:01:26.417519 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:26.417476 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37ed0b35-a2e9-49eb-95c5-793da9836c6f" path="/var/lib/kubelet/pods/37ed0b35-a2e9-49eb-95c5-793da9836c6f/volumes" Nov 04 05:01:26.418192 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:26.418171 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80a73743-0113-4f1c-94cc-38b67249b698" path="/var/lib/kubelet/pods/80a73743-0113-4f1c-94cc-38b67249b698/volumes" Nov 04 05:01:26.678231 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:26.678147 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8c59c1d-5mgkr"] Nov 04 05:01:26.678600 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:26.678584 4395 cpu_manager.go:414] "containerMap: removing container" podUID="e577d1b4-d707-40c7-a78f-c64c480c1af2" containerName="main" Nov 04 05:01:26.678600 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:26.678598 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e577d1b4-d707-40c7-a78f-c64c480c1af2" containerName="main" Nov 04 05:01:26.678772 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:26.678610 4395 cpu_manager.go:414] "containerMap: removing container" podUID="80a73743-0113-4f1c-94cc-38b67249b698" containerName="main" Nov 04 05:01:26.678772 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:26.678616 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80a73743-0113-4f1c-94cc-38b67249b698" containerName="main" Nov 04 05:01:26.678772 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:26.678622 4395 cpu_manager.go:414] "containerMap: removing container" podUID="8e40da9e-fed2-40ab-b430-9e69b8bab9d0" containerName="revoke-keys" Nov 04 05:01:26.678772 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:26.678627 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e40da9e-fed2-40ab-b430-9e69b8bab9d0" containerName="revoke-keys" Nov 04 05:01:26.678772 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:26.678670 4395 cpu_manager.go:414] "containerMap: removing container" podUID="ac06b52c-cb97-479d-813f-a9e9d6d9e67a" containerName="main" Nov 04 05:01:26.678772 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:26.678675 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac06b52c-cb97-479d-813f-a9e9d6d9e67a" containerName="main" Nov 04 05:01:26.678772 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:26.678689 4395 cpu_manager.go:414] "containerMap: removing container" podUID="37ed0b35-a2e9-49eb-95c5-793da9836c6f" containerName="main" Nov 04 05:01:26.678772 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:26.678694 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37ed0b35-a2e9-49eb-95c5-793da9836c6f" containerName="main" Nov 04 05:01:26.678772 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:26.678755 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac06b52c-cb97-479d-813f-a9e9d6d9e67a" containerName="main" Nov 04 05:01:26.678772 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:26.678765 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="37ed0b35-a2e9-49eb-95c5-793da9836c6f" containerName="main" Nov 04 05:01:26.678772 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:26.678775 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="e577d1b4-d707-40c7-a78f-c64c480c1af2" containerName="main" Nov 04 05:01:26.679342 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:26.678783 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="80a73743-0113-4f1c-94cc-38b67249b698" containerName="main" Nov 04 05:01:26.679342 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:26.678792 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e40da9e-fed2-40ab-b430-9e69b8bab9d0" containerName="revoke-keys" Nov 04 05:01:26.683781 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:26.683755 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c59c1d-5mgkr" Nov 04 05:01:26.703746 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:26.703722 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de211e54-1623-4eb4-b0b8-0c68323f373a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8c59c1d-5mgkr\" (UID: \"de211e54-1623-4eb4-b0b8-0c68323f373a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c59c1d-5mgkr" Nov 04 05:01:26.703867 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:26.703764 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srn5f\" (UniqueName: \"kubernetes.io/projected/de211e54-1623-4eb4-b0b8-0c68323f373a-kube-api-access-srn5f\") pod \"maas-api-revoke-sub-f8c59c1d-5mgkr\" (UID: \"de211e54-1623-4eb4-b0b8-0c68323f373a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c59c1d-5mgkr" Nov 04 05:01:26.703933 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:26.703884 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de211e54-1623-4eb4-b0b8-0c68323f373a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8c59c1d-5mgkr\" (UID: \"de211e54-1623-4eb4-b0b8-0c68323f373a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c59c1d-5mgkr" Nov 04 05:01:26.732439 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:26.732411 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 05:01:26.732567 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:26.732550 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nz4vh\"" Nov 04 05:01:26.782288 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:26.782257 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8c59c1d-5mgkr"] Nov 04 05:01:26.804531 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:26.804501 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de211e54-1623-4eb4-b0b8-0c68323f373a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8c59c1d-5mgkr\" (UID: \"de211e54-1623-4eb4-b0b8-0c68323f373a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c59c1d-5mgkr" Nov 04 05:01:26.804682 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:26.804546 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srn5f\" (UniqueName: \"kubernetes.io/projected/de211e54-1623-4eb4-b0b8-0c68323f373a-kube-api-access-srn5f\") pod \"maas-api-revoke-sub-f8c59c1d-5mgkr\" (UID: \"de211e54-1623-4eb4-b0b8-0c68323f373a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c59c1d-5mgkr" Nov 04 05:01:26.804682 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:26.804621 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de211e54-1623-4eb4-b0b8-0c68323f373a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8c59c1d-5mgkr\" (UID: \"de211e54-1623-4eb4-b0b8-0c68323f373a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c59c1d-5mgkr" Nov 04 05:01:26.805115 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:26.805091 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de211e54-1623-4eb4-b0b8-0c68323f373a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8c59c1d-5mgkr\" (UID: \"de211e54-1623-4eb4-b0b8-0c68323f373a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c59c1d-5mgkr" Nov 04 05:01:26.806824 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:26.806803 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de211e54-1623-4eb4-b0b8-0c68323f373a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8c59c1d-5mgkr\" (UID: \"de211e54-1623-4eb4-b0b8-0c68323f373a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c59c1d-5mgkr" Nov 04 05:01:26.863551 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:26.863525 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srn5f\" (UniqueName: \"kubernetes.io/projected/de211e54-1623-4eb4-b0b8-0c68323f373a-kube-api-access-srn5f\") pod \"maas-api-revoke-sub-f8c59c1d-5mgkr\" (UID: \"de211e54-1623-4eb4-b0b8-0c68323f373a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c59c1d-5mgkr" Nov 04 05:01:26.998726 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:26.998676 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c59c1d-5mgkr" Nov 04 05:01:27.374373 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:27.374343 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8c59c1d-5mgkr"] Nov 04 05:01:27.374556 ip-10-0-134-224 kubenswrapper[4395]: W1104 05:01:27.374531 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde211e54_1623_4eb4_b0b8_0c68323f373a.slice/crio-c2c82c8b62c2e3e4d4677494f8644e6a2bb9a71961f7ae52d1a34ddb0bb66ce9 WatchSource:0}: Error finding container c2c82c8b62c2e3e4d4677494f8644e6a2bb9a71961f7ae52d1a34ddb0bb66ce9: Status 404 returned error can't find the container with id c2c82c8b62c2e3e4d4677494f8644e6a2bb9a71961f7ae52d1a34ddb0bb66ce9 Nov 04 05:01:27.482853 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:27.482830 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-c5bb6d/distinct-2-w2-c5bb6d-kserve-78dff78654-r524h"] Nov 04 05:01:27.487802 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:27.487784 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c5bb6d/distinct-2-w2-c5bb6d-kserve-78dff78654-r524h" Nov 04 05:01:27.509520 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:27.509487 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f554177-f75c-438e-8d57-71fdd73f7d2e-model-cache\") pod \"distinct-2-w2-c5bb6d-kserve-78dff78654-r524h\" (UID: \"8f554177-f75c-438e-8d57-71fdd73f7d2e\") " pod="e2e-models-e2e-worker-w2-c5bb6d/distinct-2-w2-c5bb6d-kserve-78dff78654-r524h" Nov 04 05:01:27.509637 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:27.509550 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f554177-f75c-438e-8d57-71fdd73f7d2e-tls-certs\") pod \"distinct-2-w2-c5bb6d-kserve-78dff78654-r524h\" (UID: \"8f554177-f75c-438e-8d57-71fdd73f7d2e\") " pod="e2e-models-e2e-worker-w2-c5bb6d/distinct-2-w2-c5bb6d-kserve-78dff78654-r524h" Nov 04 05:01:27.509701 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:27.509660 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv4mj\" (UniqueName: \"kubernetes.io/projected/8f554177-f75c-438e-8d57-71fdd73f7d2e-kube-api-access-lv4mj\") pod \"distinct-2-w2-c5bb6d-kserve-78dff78654-r524h\" (UID: \"8f554177-f75c-438e-8d57-71fdd73f7d2e\") " pod="e2e-models-e2e-worker-w2-c5bb6d/distinct-2-w2-c5bb6d-kserve-78dff78654-r524h" Nov 04 05:01:27.509701 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:27.509689 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f554177-f75c-438e-8d57-71fdd73f7d2e-home\") pod \"distinct-2-w2-c5bb6d-kserve-78dff78654-r524h\" (UID: \"8f554177-f75c-438e-8d57-71fdd73f7d2e\") " pod="e2e-models-e2e-worker-w2-c5bb6d/distinct-2-w2-c5bb6d-kserve-78dff78654-r524h" Nov 04 05:01:27.509825 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:27.509806 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f554177-f75c-438e-8d57-71fdd73f7d2e-dshm\") pod \"distinct-2-w2-c5bb6d-kserve-78dff78654-r524h\" (UID: \"8f554177-f75c-438e-8d57-71fdd73f7d2e\") " pod="e2e-models-e2e-worker-w2-c5bb6d/distinct-2-w2-c5bb6d-kserve-78dff78654-r524h" Nov 04 05:01:27.509869 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:27.509850 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f554177-f75c-438e-8d57-71fdd73f7d2e-tmp-dir\") pod \"distinct-2-w2-c5bb6d-kserve-78dff78654-r524h\" (UID: \"8f554177-f75c-438e-8d57-71fdd73f7d2e\") " pod="e2e-models-e2e-worker-w2-c5bb6d/distinct-2-w2-c5bb6d-kserve-78dff78654-r524h" Nov 04 05:01:27.536540 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:27.536512 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-c5bb6d\"/\"openshift-service-ca.crt\"" Nov 04 05:01:27.582744 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:27.582721 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c5bb6d\"/\"distinct-2-w2-c5bb6d-kserve-self-signed-certs\"" Nov 04 05:01:27.582931 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:27.582726 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c5bb6d\"/\"default-dockercfg-kcsrz\"" Nov 04 05:01:27.583049 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:27.582772 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-c5bb6d\"/\"kube-root-ca.crt\"" Nov 04 05:01:27.586947 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:27.586919 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c5bb6d/distinct-2-w2-c5bb6d-kserve-78dff78654-r524h"] Nov 04 05:01:27.610658 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:27.610632 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f554177-f75c-438e-8d57-71fdd73f7d2e-model-cache\") pod \"distinct-2-w2-c5bb6d-kserve-78dff78654-r524h\" (UID: \"8f554177-f75c-438e-8d57-71fdd73f7d2e\") " pod="e2e-models-e2e-worker-w2-c5bb6d/distinct-2-w2-c5bb6d-kserve-78dff78654-r524h" Nov 04 05:01:27.610773 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:27.610709 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f554177-f75c-438e-8d57-71fdd73f7d2e-tls-certs\") pod \"distinct-2-w2-c5bb6d-kserve-78dff78654-r524h\" (UID: \"8f554177-f75c-438e-8d57-71fdd73f7d2e\") " pod="e2e-models-e2e-worker-w2-c5bb6d/distinct-2-w2-c5bb6d-kserve-78dff78654-r524h" Nov 04 05:01:27.610836 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:27.610771 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv4mj\" (UniqueName: \"kubernetes.io/projected/8f554177-f75c-438e-8d57-71fdd73f7d2e-kube-api-access-lv4mj\") pod \"distinct-2-w2-c5bb6d-kserve-78dff78654-r524h\" (UID: \"8f554177-f75c-438e-8d57-71fdd73f7d2e\") " pod="e2e-models-e2e-worker-w2-c5bb6d/distinct-2-w2-c5bb6d-kserve-78dff78654-r524h" Nov 04 05:01:27.610836 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:27.610799 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f554177-f75c-438e-8d57-71fdd73f7d2e-home\") pod \"distinct-2-w2-c5bb6d-kserve-78dff78654-r524h\" (UID: \"8f554177-f75c-438e-8d57-71fdd73f7d2e\") " pod="e2e-models-e2e-worker-w2-c5bb6d/distinct-2-w2-c5bb6d-kserve-78dff78654-r524h" Nov 04 05:01:27.610938 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:27.610909 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f554177-f75c-438e-8d57-71fdd73f7d2e-dshm\") pod \"distinct-2-w2-c5bb6d-kserve-78dff78654-r524h\" (UID: \"8f554177-f75c-438e-8d57-71fdd73f7d2e\") " pod="e2e-models-e2e-worker-w2-c5bb6d/distinct-2-w2-c5bb6d-kserve-78dff78654-r524h" Nov 04 05:01:27.611078 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:27.611000 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f554177-f75c-438e-8d57-71fdd73f7d2e-tmp-dir\") pod \"distinct-2-w2-c5bb6d-kserve-78dff78654-r524h\" (UID: \"8f554177-f75c-438e-8d57-71fdd73f7d2e\") " pod="e2e-models-e2e-worker-w2-c5bb6d/distinct-2-w2-c5bb6d-kserve-78dff78654-r524h" Nov 04 05:01:27.611265 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:27.611242 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f554177-f75c-438e-8d57-71fdd73f7d2e-model-cache\") pod \"distinct-2-w2-c5bb6d-kserve-78dff78654-r524h\" (UID: \"8f554177-f75c-438e-8d57-71fdd73f7d2e\") " pod="e2e-models-e2e-worker-w2-c5bb6d/distinct-2-w2-c5bb6d-kserve-78dff78654-r524h" Nov 04 05:01:27.611396 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:27.611319 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f554177-f75c-438e-8d57-71fdd73f7d2e-home\") pod \"distinct-2-w2-c5bb6d-kserve-78dff78654-r524h\" (UID: \"8f554177-f75c-438e-8d57-71fdd73f7d2e\") " pod="e2e-models-e2e-worker-w2-c5bb6d/distinct-2-w2-c5bb6d-kserve-78dff78654-r524h" Nov 04 05:01:27.611396 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:27.611373 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f554177-f75c-438e-8d57-71fdd73f7d2e-tmp-dir\") pod \"distinct-2-w2-c5bb6d-kserve-78dff78654-r524h\" (UID: \"8f554177-f75c-438e-8d57-71fdd73f7d2e\") " pod="e2e-models-e2e-worker-w2-c5bb6d/distinct-2-w2-c5bb6d-kserve-78dff78654-r524h" Nov 04 05:01:27.613152 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:27.613130 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f554177-f75c-438e-8d57-71fdd73f7d2e-dshm\") pod \"distinct-2-w2-c5bb6d-kserve-78dff78654-r524h\" (UID: \"8f554177-f75c-438e-8d57-71fdd73f7d2e\") " pod="e2e-models-e2e-worker-w2-c5bb6d/distinct-2-w2-c5bb6d-kserve-78dff78654-r524h" Nov 04 05:01:27.613854 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:27.613834 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f554177-f75c-438e-8d57-71fdd73f7d2e-tls-certs\") pod \"distinct-2-w2-c5bb6d-kserve-78dff78654-r524h\" (UID: \"8f554177-f75c-438e-8d57-71fdd73f7d2e\") " pod="e2e-models-e2e-worker-w2-c5bb6d/distinct-2-w2-c5bb6d-kserve-78dff78654-r524h" Nov 04 05:01:27.669088 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:27.669058 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv4mj\" (UniqueName: \"kubernetes.io/projected/8f554177-f75c-438e-8d57-71fdd73f7d2e-kube-api-access-lv4mj\") pod \"distinct-2-w2-c5bb6d-kserve-78dff78654-r524h\" (UID: \"8f554177-f75c-438e-8d57-71fdd73f7d2e\") " pod="e2e-models-e2e-worker-w2-c5bb6d/distinct-2-w2-c5bb6d-kserve-78dff78654-r524h" Nov 04 05:01:27.798787 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:27.798761 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c5bb6d/distinct-2-w2-c5bb6d-kserve-78dff78654-r524h" Nov 04 05:01:27.914586 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:27.914545 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c59c1d-5mgkr" event={"ID":"de211e54-1623-4eb4-b0b8-0c68323f373a","Type":"ContainerStarted","Data":"52684c85894525504977cb53047f2e22222e2d45f01e6f002204b48e1eac0238"} Nov 04 05:01:27.914727 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:27.914618 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c59c1d-5mgkr" event={"ID":"de211e54-1623-4eb4-b0b8-0c68323f373a","Type":"ContainerStarted","Data":"c2c82c8b62c2e3e4d4677494f8644e6a2bb9a71961f7ae52d1a34ddb0bb66ce9"} Nov 04 05:01:28.051917 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:28.051870 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c59c1d-5mgkr" podStartSLOduration=2.051856497 podStartE2EDuration="2.051856497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:01:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:01:28.051032417 +0000 UTC m=+929.741114887" watchObservedRunningTime="2026-11-04 05:01:28.051856497 +0000 UTC m=+929.741938980" Nov 04 05:01:28.187794 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:28.187755 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c5bb6d/distinct-2-w2-c5bb6d-kserve-78dff78654-r524h"] Nov 04 05:01:28.928740 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:28.928707 4395 generic.go:356] "Generic (PLEG): container finished" podID="de211e54-1623-4eb4-b0b8-0c68323f373a" containerID="52684c85894525504977cb53047f2e22222e2d45f01e6f002204b48e1eac0238" exitCode=0 Nov 04 05:01:28.929202 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:28.928788 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c59c1d-5mgkr" event={"ID":"de211e54-1623-4eb4-b0b8-0c68323f373a","Type":"ContainerDied","Data":"52684c85894525504977cb53047f2e22222e2d45f01e6f002204b48e1eac0238"} Nov 04 05:01:28.930523 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:28.930496 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c5bb6d/distinct-2-w2-c5bb6d-kserve-78dff78654-r524h" event={"ID":"8f554177-f75c-438e-8d57-71fdd73f7d2e","Type":"ContainerStarted","Data":"4bb152bc105e6af21a730417de47d0cba974262ca855b8dfe99f7e3ed51bf453"} Nov 04 05:01:28.930634 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:28.930536 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c5bb6d/distinct-2-w2-c5bb6d-kserve-78dff78654-r524h" event={"ID":"8f554177-f75c-438e-8d57-71fdd73f7d2e","Type":"ContainerStarted","Data":"2d9279d7a2b0514f7f0db8f384b7c7810947ce5d050166029166f794209bcef5"} Nov 04 05:01:29.012939 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:29.012908 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-19bf5b-7c6d784c4-s8rpn" podUID="5368507f-308f-447a-8da1-f6fc28c7a88d" containerName="payload-processing" containerID="cri-o://f4dea3bbdf37275210e082ce77cb010cec001b882af44c0ad0ee269f6661549f" gracePeriod=25 Nov 04 05:01:29.148234 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:29.148183 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-c5bb6d/distinct-2-w2-c5bb6d-kserve-78dff78654-r524h" podStartSLOduration=2.148169245 podStartE2EDuration="2.148169245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:01:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:01:29.145671046 +0000 UTC m=+930.835753492" watchObservedRunningTime="2026-11-04 05:01:29.148169245 +0000 UTC m=+930.838251692" Nov 04 05:01:29.799114 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:29.799080 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c5bb6d/distinct-2-w2-c5bb6d-kserve-78dff78654-r524h" Nov 04 05:01:30.094827 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:30.094800 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c59c1d-5mgkr" Nov 04 05:01:30.133856 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:30.133830 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de211e54-1623-4eb4-b0b8-0c68323f373a-kube-api-access-srn5f\" (UniqueName: \"kubernetes.io/projected/de211e54-1623-4eb4-b0b8-0c68323f373a-kube-api-access-srn5f\") pod \"de211e54-1623-4eb4-b0b8-0c68323f373a\" (UID: \"de211e54-1623-4eb4-b0b8-0c68323f373a\") " Nov 04 05:01:30.133978 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:30.133893 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de211e54-1623-4eb4-b0b8-0c68323f373a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de211e54-1623-4eb4-b0b8-0c68323f373a-maas-api-xks-ca\") pod \"de211e54-1623-4eb4-b0b8-0c68323f373a\" (UID: \"de211e54-1623-4eb4-b0b8-0c68323f373a\") " Nov 04 05:01:30.133978 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:30.133972 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de211e54-1623-4eb4-b0b8-0c68323f373a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de211e54-1623-4eb4-b0b8-0c68323f373a-maas-api-openshift-ca\") pod \"de211e54-1623-4eb4-b0b8-0c68323f373a\" (UID: \"de211e54-1623-4eb4-b0b8-0c68323f373a\") " Nov 04 05:01:30.134384 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:30.134355 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de211e54-1623-4eb4-b0b8-0c68323f373a-maas-api-openshift-ca" pod "de211e54-1623-4eb4-b0b8-0c68323f373a" (UID: "de211e54-1623-4eb4-b0b8-0c68323f373a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:01:30.135878 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:30.135855 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de211e54-1623-4eb4-b0b8-0c68323f373a-kube-api-access-srn5f" pod "de211e54-1623-4eb4-b0b8-0c68323f373a" (UID: "de211e54-1623-4eb4-b0b8-0c68323f373a"). InnerVolumeSpecName "kube-api-access-srn5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:01:30.135878 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:30.135866 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de211e54-1623-4eb4-b0b8-0c68323f373a-maas-api-xks-ca" pod "de211e54-1623-4eb4-b0b8-0c68323f373a" (UID: "de211e54-1623-4eb4-b0b8-0c68323f373a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:01:30.234929 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:30.234901 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de211e54-1623-4eb4-b0b8-0c68323f373a-maas-api-openshift-ca\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:30.234929 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:30.234927 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srn5f\" (UniqueName: \"kubernetes.io/projected/de211e54-1623-4eb4-b0b8-0c68323f373a-kube-api-access-srn5f\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:30.235091 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:30.234937 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de211e54-1623-4eb4-b0b8-0c68323f373a-maas-api-xks-ca\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:30.943161 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:30.943132 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c59c1d-5mgkr" Nov 04 05:01:30.943353 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:30.943167 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8c59c1d-5mgkr" event={"ID":"de211e54-1623-4eb4-b0b8-0c68323f373a","Type":"ContainerDied","Data":"c2c82c8b62c2e3e4d4677494f8644e6a2bb9a71961f7ae52d1a34ddb0bb66ce9"} Nov 04 05:01:30.943353 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:30.943216 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2c82c8b62c2e3e4d4677494f8644e6a2bb9a71961f7ae52d1a34ddb0bb66ce9" Nov 04 05:01:31.550635 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:31.550596 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-19bf5b-7c6d784c4-s8rpn" podUID="5368507f-308f-447a-8da1-f6fc28c7a88d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 05:01:33.457743 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:01:33.457707 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 05:01:34.715768 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:34.715737 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-db8fae-856fc89fc8-x97fb"] Nov 04 05:01:37.799673 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:37.799639 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-c5bb6d/distinct-2-w2-c5bb6d-kserve-78dff78654-r524h" Nov 04 05:01:37.812745 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:37.812718 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-c5bb6d/distinct-2-w2-c5bb6d-kserve-78dff78654-r524h" Nov 04 05:01:37.967002 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:37.966961 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-db8fae/premium-w6-db8fae-kserve-6c464dbc8b-xvnsz" podUID="002e54f5-f516-44b1-8145-a26084b77264" containerName="main" containerID="cri-o://c08e3ddc635f2d165c9975db699524b2ffa69df63a32d4d8234a6104ed91f3f7" gracePeriod=45 Nov 04 05:01:38.008909 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:38.008880 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c5bb6d/distinct-2-w2-c5bb6d-kserve-78dff78654-r524h" Nov 04 05:01:38.036414 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:38.036382 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-db8fae/premium-w6-db8fae-kserve-6c464dbc8b-xvnsz" podUID="002e54f5-f516-44b1-8145-a26084b77264" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": dial tcp 10.134.0.56:8000: connect: connection refused" Nov 04 05:01:38.208354 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:38.208332 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-db8fae/premium-w6-db8fae-kserve-6c464dbc8b-xvnsz" Nov 04 05:01:38.315633 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:38.315600 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/002e54f5-f516-44b1-8145-a26084b77264-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/002e54f5-f516-44b1-8145-a26084b77264-tmp-dir\") pod \"002e54f5-f516-44b1-8145-a26084b77264\" (UID: \"002e54f5-f516-44b1-8145-a26084b77264\") " Nov 04 05:01:38.315633 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:38.315638 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/002e54f5-f516-44b1-8145-a26084b77264-home\" (UniqueName: \"kubernetes.io/empty-dir/002e54f5-f516-44b1-8145-a26084b77264-home\") pod \"002e54f5-f516-44b1-8145-a26084b77264\" (UID: \"002e54f5-f516-44b1-8145-a26084b77264\") " Nov 04 05:01:38.315864 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:38.315737 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/002e54f5-f516-44b1-8145-a26084b77264-tls-certs\" (UniqueName: \"kubernetes.io/secret/002e54f5-f516-44b1-8145-a26084b77264-tls-certs\") pod \"002e54f5-f516-44b1-8145-a26084b77264\" (UID: \"002e54f5-f516-44b1-8145-a26084b77264\") " Nov 04 05:01:38.315864 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:38.315764 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/002e54f5-f516-44b1-8145-a26084b77264-dshm\" (UniqueName: \"kubernetes.io/empty-dir/002e54f5-f516-44b1-8145-a26084b77264-dshm\") pod \"002e54f5-f516-44b1-8145-a26084b77264\" (UID: \"002e54f5-f516-44b1-8145-a26084b77264\") " Nov 04 05:01:38.315864 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:38.315812 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/002e54f5-f516-44b1-8145-a26084b77264-kube-api-access-jm846\" (UniqueName: \"kubernetes.io/projected/002e54f5-f516-44b1-8145-a26084b77264-kube-api-access-jm846\") pod \"002e54f5-f516-44b1-8145-a26084b77264\" (UID: \"002e54f5-f516-44b1-8145-a26084b77264\") " Nov 04 05:01:38.315864 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:38.315857 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/002e54f5-f516-44b1-8145-a26084b77264-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/002e54f5-f516-44b1-8145-a26084b77264-model-cache\") pod \"002e54f5-f516-44b1-8145-a26084b77264\" (UID: \"002e54f5-f516-44b1-8145-a26084b77264\") " Nov 04 05:01:38.316081 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:38.315859 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/002e54f5-f516-44b1-8145-a26084b77264-tmp-dir" pod "002e54f5-f516-44b1-8145-a26084b77264" (UID: "002e54f5-f516-44b1-8145-a26084b77264"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:01:38.316081 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:38.315952 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/002e54f5-f516-44b1-8145-a26084b77264-home" pod "002e54f5-f516-44b1-8145-a26084b77264" (UID: "002e54f5-f516-44b1-8145-a26084b77264"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:01:38.316217 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:38.316194 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/002e54f5-f516-44b1-8145-a26084b77264-model-cache" pod "002e54f5-f516-44b1-8145-a26084b77264" (UID: "002e54f5-f516-44b1-8145-a26084b77264"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:01:38.316275 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:38.316218 4395 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/002e54f5-f516-44b1-8145-a26084b77264-tmp-dir\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:38.316275 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:38.316236 4395 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/002e54f5-f516-44b1-8145-a26084b77264-home\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:38.318577 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:38.318242 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/002e54f5-f516-44b1-8145-a26084b77264-kube-api-access-jm846" pod "002e54f5-f516-44b1-8145-a26084b77264" (UID: "002e54f5-f516-44b1-8145-a26084b77264"). InnerVolumeSpecName "kube-api-access-jm846". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:01:38.318577 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:38.318319 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/002e54f5-f516-44b1-8145-a26084b77264-tls-certs" pod "002e54f5-f516-44b1-8145-a26084b77264" (UID: "002e54f5-f516-44b1-8145-a26084b77264"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:01:38.318577 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:38.318410 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/002e54f5-f516-44b1-8145-a26084b77264-dshm" pod "002e54f5-f516-44b1-8145-a26084b77264" (UID: "002e54f5-f516-44b1-8145-a26084b77264"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:01:38.416783 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:38.416690 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm846\" (UniqueName: \"kubernetes.io/projected/002e54f5-f516-44b1-8145-a26084b77264-kube-api-access-jm846\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:38.416783 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:38.416719 4395 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/002e54f5-f516-44b1-8145-a26084b77264-model-cache\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:38.416783 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:38.416730 4395 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/002e54f5-f516-44b1-8145-a26084b77264-tls-certs\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:38.416783 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:38.416738 4395 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/002e54f5-f516-44b1-8145-a26084b77264-dshm\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:38.998782 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:38.998694 4395 generic.go:356] "Generic (PLEG): container finished" podID="002e54f5-f516-44b1-8145-a26084b77264" containerID="c08e3ddc635f2d165c9975db699524b2ffa69df63a32d4d8234a6104ed91f3f7" exitCode=0 Nov 04 05:01:38.998782 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:38.998756 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-db8fae/premium-w6-db8fae-kserve-6c464dbc8b-xvnsz" Nov 04 05:01:38.999327 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:38.998784 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-db8fae/premium-w6-db8fae-kserve-6c464dbc8b-xvnsz" event={"ID":"002e54f5-f516-44b1-8145-a26084b77264","Type":"ContainerDied","Data":"c08e3ddc635f2d165c9975db699524b2ffa69df63a32d4d8234a6104ed91f3f7"} Nov 04 05:01:38.999327 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:38.998834 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-db8fae/premium-w6-db8fae-kserve-6c464dbc8b-xvnsz" event={"ID":"002e54f5-f516-44b1-8145-a26084b77264","Type":"ContainerDied","Data":"5fc52f0ef4d5e73fd100fd44aaffc1abec85794a4f2aaba9b9ed69db996f853e"} Nov 04 05:01:38.999327 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:38.998858 4395 scope.go:122] "RemoveContainer" containerID="c08e3ddc635f2d165c9975db699524b2ffa69df63a32d4d8234a6104ed91f3f7" Nov 04 05:01:39.009019 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:39.008989 4395 scope.go:122] "RemoveContainer" containerID="c08e3ddc635f2d165c9975db699524b2ffa69df63a32d4d8234a6104ed91f3f7" Nov 04 05:01:39.009290 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:01:39.009269 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c08e3ddc635f2d165c9975db699524b2ffa69df63a32d4d8234a6104ed91f3f7\": container with ID starting with c08e3ddc635f2d165c9975db699524b2ffa69df63a32d4d8234a6104ed91f3f7 not found: ID does not exist" containerID="c08e3ddc635f2d165c9975db699524b2ffa69df63a32d4d8234a6104ed91f3f7" Nov 04 05:01:39.009388 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:39.009314 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c08e3ddc635f2d165c9975db699524b2ffa69df63a32d4d8234a6104ed91f3f7"} err="failed to get container status \"c08e3ddc635f2d165c9975db699524b2ffa69df63a32d4d8234a6104ed91f3f7\": rpc error: code = NotFound desc = could not find container \"c08e3ddc635f2d165c9975db699524b2ffa69df63a32d4d8234a6104ed91f3f7\": container with ID starting with c08e3ddc635f2d165c9975db699524b2ffa69df63a32d4d8234a6104ed91f3f7 not found: ID does not exist" Nov 04 05:01:39.151137 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:39.151108 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-db8fae/premium-w6-db8fae-kserve-6c464dbc8b-xvnsz"] Nov 04 05:01:39.196925 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:39.196905 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-db8fae/premium-w6-db8fae-kserve-6c464dbc8b-xvnsz"] Nov 04 05:01:39.716946 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:39.716882 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-db8fae-856fc89fc8-x97fb" podUID="24037ab8-fddf-4dd8-b170-96bdc4279e3f" containerName="payload-processing" containerID="cri-o://106d718d23426bfbae521a1071b7b9bffd115cf74f6a1b71cae79af7f101192e" gracePeriod=25 Nov 04 05:01:39.825749 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:39.825719 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-204e38/distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz"] Nov 04 05:01:39.826162 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:39.826146 4395 cpu_manager.go:414] "containerMap: removing container" podUID="002e54f5-f516-44b1-8145-a26084b77264" containerName="main" Nov 04 05:01:39.826162 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:39.826159 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="002e54f5-f516-44b1-8145-a26084b77264" containerName="main" Nov 04 05:01:39.826315 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:39.826183 4395 cpu_manager.go:414] "containerMap: removing container" podUID="de211e54-1623-4eb4-b0b8-0c68323f373a" containerName="revoke-keys" Nov 04 05:01:39.826315 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:39.826188 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de211e54-1623-4eb4-b0b8-0c68323f373a" containerName="revoke-keys" Nov 04 05:01:39.826315 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:39.826248 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="de211e54-1623-4eb4-b0b8-0c68323f373a" containerName="revoke-keys" Nov 04 05:01:39.826315 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:39.826261 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="002e54f5-f516-44b1-8145-a26084b77264" containerName="main" Nov 04 05:01:39.829612 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:39.829594 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-204e38/distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz" Nov 04 05:01:39.880009 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:39.878721 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-204e38\"/\"kube-root-ca.crt\"" Nov 04 05:01:39.897269 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:39.896460 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-204e38\"/\"default-dockercfg-wlp5v\"" Nov 04 05:01:39.897269 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:39.896512 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-204e38\"/\"openshift-service-ca.crt\"" Nov 04 05:01:39.930809 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:39.927847 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-204e38\"/\"distinct-2-w3-204e38-kserve-self-signed-certs\"" Nov 04 05:01:39.930809 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:39.930746 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-204e38/distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz"] Nov 04 05:01:39.932977 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:39.932949 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-tmp-dir\") pod \"distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz\" (UID: \"6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1\") " pod="e2e-models-e2e-worker-w3-204e38/distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz" Nov 04 05:01:39.933103 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:39.933005 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpqsl\" (UniqueName: \"kubernetes.io/projected/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-kube-api-access-fpqsl\") pod \"distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz\" (UID: \"6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1\") " pod="e2e-models-e2e-worker-w3-204e38/distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz" Nov 04 05:01:39.933103 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:39.933041 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-dshm\") pod \"distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz\" (UID: \"6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1\") " pod="e2e-models-e2e-worker-w3-204e38/distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz" Nov 04 05:01:39.933227 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:39.933117 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-home\") pod \"distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz\" (UID: \"6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1\") " pod="e2e-models-e2e-worker-w3-204e38/distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz" Nov 04 05:01:39.933227 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:39.933209 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-tls-certs\") pod \"distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz\" (UID: \"6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1\") " pod="e2e-models-e2e-worker-w3-204e38/distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz" Nov 04 05:01:39.933350 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:39.933254 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-model-cache\") pod \"distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz\" (UID: \"6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1\") " pod="e2e-models-e2e-worker-w3-204e38/distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz" Nov 04 05:01:40.007835 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:40.007807 4395 generic.go:356] "Generic (PLEG): container finished" podID="24037ab8-fddf-4dd8-b170-96bdc4279e3f" containerID="106d718d23426bfbae521a1071b7b9bffd115cf74f6a1b71cae79af7f101192e" exitCode=0 Nov 04 05:01:40.008214 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:40.007874 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-db8fae-856fc89fc8-x97fb" event={"ID":"24037ab8-fddf-4dd8-b170-96bdc4279e3f","Type":"ContainerDied","Data":"106d718d23426bfbae521a1071b7b9bffd115cf74f6a1b71cae79af7f101192e"} Nov 04 05:01:40.008214 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:40.007923 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-db8fae-856fc89fc8-x97fb" event={"ID":"24037ab8-fddf-4dd8-b170-96bdc4279e3f","Type":"ContainerDied","Data":"2e216106bafc503a128524e7ca87146a45ee7da856d75fe3cebf3896ee124ffa"} Nov 04 05:01:40.008214 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:40.007939 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e216106bafc503a128524e7ca87146a45ee7da856d75fe3cebf3896ee124ffa" Nov 04 05:01:40.017626 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:40.017609 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-db8fae-856fc89fc8-x97fb" Nov 04 05:01:40.034735 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:40.034712 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-tls-certs\") pod \"distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz\" (UID: \"6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1\") " pod="e2e-models-e2e-worker-w3-204e38/distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz" Nov 04 05:01:40.034834 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:40.034751 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-model-cache\") pod \"distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz\" (UID: \"6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1\") " pod="e2e-models-e2e-worker-w3-204e38/distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz" Nov 04 05:01:40.034834 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:40.034799 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-tmp-dir\") pod \"distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz\" (UID: \"6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1\") " pod="e2e-models-e2e-worker-w3-204e38/distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz" Nov 04 05:01:40.034926 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:40.034841 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpqsl\" (UniqueName: \"kubernetes.io/projected/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-kube-api-access-fpqsl\") pod \"distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz\" (UID: \"6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1\") " pod="e2e-models-e2e-worker-w3-204e38/distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz" Nov 04 05:01:40.034926 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:40.034874 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-dshm\") pod \"distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz\" (UID: \"6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1\") " pod="e2e-models-e2e-worker-w3-204e38/distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz" Nov 04 05:01:40.035024 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:40.034948 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-home\") pod \"distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz\" (UID: \"6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1\") " pod="e2e-models-e2e-worker-w3-204e38/distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz" Nov 04 05:01:40.035282 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:40.035256 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-tmp-dir\") pod \"distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz\" (UID: \"6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1\") " pod="e2e-models-e2e-worker-w3-204e38/distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz" Nov 04 05:01:40.035398 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:40.035291 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-model-cache\") pod \"distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz\" (UID: \"6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1\") " pod="e2e-models-e2e-worker-w3-204e38/distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz" Nov 04 05:01:40.035398 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:40.035338 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-home\") pod \"distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz\" (UID: \"6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1\") " pod="e2e-models-e2e-worker-w3-204e38/distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz" Nov 04 05:01:40.037469 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:40.037450 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-dshm\") pod \"distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz\" (UID: \"6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1\") " pod="e2e-models-e2e-worker-w3-204e38/distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz" Nov 04 05:01:40.037545 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:40.037457 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-tls-certs\") pod \"distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz\" (UID: \"6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1\") " pod="e2e-models-e2e-worker-w3-204e38/distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz" Nov 04 05:01:40.094655 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:40.094630 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpqsl\" (UniqueName: \"kubernetes.io/projected/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-kube-api-access-fpqsl\") pod \"distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz\" (UID: \"6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1\") " pod="e2e-models-e2e-worker-w3-204e38/distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz" Nov 04 05:01:40.135312 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:40.135280 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24037ab8-fddf-4dd8-b170-96bdc4279e3f-kube-api-access-btmvd\" (UniqueName: \"kubernetes.io/projected/24037ab8-fddf-4dd8-b170-96bdc4279e3f-kube-api-access-btmvd\") pod \"24037ab8-fddf-4dd8-b170-96bdc4279e3f\" (UID: \"24037ab8-fddf-4dd8-b170-96bdc4279e3f\") " Nov 04 05:01:40.135426 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:40.135368 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24037ab8-fddf-4dd8-b170-96bdc4279e3f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24037ab8-fddf-4dd8-b170-96bdc4279e3f-plugins-config-volume\") pod \"24037ab8-fddf-4dd8-b170-96bdc4279e3f\" (UID: \"24037ab8-fddf-4dd8-b170-96bdc4279e3f\") " Nov 04 05:01:40.135770 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:40.135743 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24037ab8-fddf-4dd8-b170-96bdc4279e3f-plugins-config-volume" pod "24037ab8-fddf-4dd8-b170-96bdc4279e3f" (UID: "24037ab8-fddf-4dd8-b170-96bdc4279e3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:01:40.137800 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:40.137769 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24037ab8-fddf-4dd8-b170-96bdc4279e3f-kube-api-access-btmvd" pod "24037ab8-fddf-4dd8-b170-96bdc4279e3f" (UID: "24037ab8-fddf-4dd8-b170-96bdc4279e3f"). InnerVolumeSpecName "kube-api-access-btmvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:01:40.141893 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:40.141871 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-204e38/distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz" Nov 04 05:01:40.236945 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:40.236428 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btmvd\" (UniqueName: \"kubernetes.io/projected/24037ab8-fddf-4dd8-b170-96bdc4279e3f-kube-api-access-btmvd\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:40.236945 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:40.236456 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24037ab8-fddf-4dd8-b170-96bdc4279e3f-plugins-config-volume\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:40.418066 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:40.418039 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="002e54f5-f516-44b1-8145-a26084b77264" path="/var/lib/kubelet/pods/002e54f5-f516-44b1-8145-a26084b77264/volumes" Nov 04 05:01:40.542807 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:40.542782 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-204e38/distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz"] Nov 04 05:01:40.543756 ip-10-0-134-224 kubenswrapper[4395]: W1104 05:01:40.543729 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ecdc46c_07cd_4c9e_af06_18c6ed1c39f1.slice/crio-a21b42ba6148bb027b165803d79611a29520621311f72a05e3425e5da90dcd5a WatchSource:0}: Error finding container a21b42ba6148bb027b165803d79611a29520621311f72a05e3425e5da90dcd5a: Status 404 returned error can't find the container with id a21b42ba6148bb027b165803d79611a29520621311f72a05e3425e5da90dcd5a Nov 04 05:01:41.016753 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:41.016667 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-204e38/distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz" event={"ID":"6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1","Type":"ContainerStarted","Data":"b4107181df7b48600429c85710c61f571d0912a131c5232e70ab36117d2b1900"} Nov 04 05:01:41.016753 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:41.016719 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-204e38/distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz" event={"ID":"6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1","Type":"ContainerStarted","Data":"a21b42ba6148bb027b165803d79611a29520621311f72a05e3425e5da90dcd5a"} Nov 04 05:01:41.016753 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:41.016745 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-db8fae-856fc89fc8-x97fb" Nov 04 05:01:41.142926 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:41.142896 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-204e38/distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz" Nov 04 05:01:41.175864 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:41.175824 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-204e38/distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz" podStartSLOduration=2.175812291 podStartE2EDuration="2.175812291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:01:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:01:41.173768763 +0000 UTC m=+942.863851207" watchObservedRunningTime="2026-11-04 05:01:41.175812291 +0000 UTC m=+942.865894769" Nov 04 05:01:41.278211 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:41.278147 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-db8fae-856fc89fc8-x97fb"] Nov 04 05:01:41.281676 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:41.281653 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-db8fae-856fc89fc8-x97fb"] Nov 04 05:01:41.550499 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:41.550458 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-19bf5b-7c6d784c4-s8rpn" podUID="5368507f-308f-447a-8da1-f6fc28c7a88d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 05:01:42.420751 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:42.420718 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24037ab8-fddf-4dd8-b170-96bdc4279e3f" path="/var/lib/kubelet/pods/24037ab8-fddf-4dd8-b170-96bdc4279e3f/volumes" Nov 04 05:01:49.358320 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:49.355406 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-19bf5b-7c6d784c4-s8rpn" Nov 04 05:01:49.435320 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:49.434351 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5368507f-308f-447a-8da1-f6fc28c7a88d-kube-api-access-854x4\" (UniqueName: \"kubernetes.io/projected/5368507f-308f-447a-8da1-f6fc28c7a88d-kube-api-access-854x4\") pod \"5368507f-308f-447a-8da1-f6fc28c7a88d\" (UID: \"5368507f-308f-447a-8da1-f6fc28c7a88d\") " Nov 04 05:01:49.435320 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:49.434466 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5368507f-308f-447a-8da1-f6fc28c7a88d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5368507f-308f-447a-8da1-f6fc28c7a88d-plugins-config-volume\") pod \"5368507f-308f-447a-8da1-f6fc28c7a88d\" (UID: \"5368507f-308f-447a-8da1-f6fc28c7a88d\") " Nov 04 05:01:49.435585 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:49.435499 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5368507f-308f-447a-8da1-f6fc28c7a88d-plugins-config-volume" pod "5368507f-308f-447a-8da1-f6fc28c7a88d" (UID: "5368507f-308f-447a-8da1-f6fc28c7a88d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:01:49.437360 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:49.437319 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5368507f-308f-447a-8da1-f6fc28c7a88d-kube-api-access-854x4" pod "5368507f-308f-447a-8da1-f6fc28c7a88d" (UID: "5368507f-308f-447a-8da1-f6fc28c7a88d"). InnerVolumeSpecName "kube-api-access-854x4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:01:49.536735 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:49.536321 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-854x4\" (UniqueName: \"kubernetes.io/projected/5368507f-308f-447a-8da1-f6fc28c7a88d-kube-api-access-854x4\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:49.536735 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:49.536358 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5368507f-308f-447a-8da1-f6fc28c7a88d-plugins-config-volume\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:50.080140 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:50.080026 4395 generic.go:356] "Generic (PLEG): container finished" podID="5368507f-308f-447a-8da1-f6fc28c7a88d" containerID="f4dea3bbdf37275210e082ce77cb010cec001b882af44c0ad0ee269f6661549f" exitCode=0 Nov 04 05:01:50.080553 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:50.080486 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-19bf5b-7c6d784c4-s8rpn" event={"ID":"5368507f-308f-447a-8da1-f6fc28c7a88d","Type":"ContainerDied","Data":"f4dea3bbdf37275210e082ce77cb010cec001b882af44c0ad0ee269f6661549f"} Nov 04 05:01:50.080768 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:50.080710 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-19bf5b-7c6d784c4-s8rpn" event={"ID":"5368507f-308f-447a-8da1-f6fc28c7a88d","Type":"ContainerDied","Data":"7a5057fd640abb02713712d52a27d552bce26a5c04428b1b63f2562bb0709dad"} Nov 04 05:01:50.080952 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:50.080881 4395 scope.go:122] "RemoveContainer" containerID="f4dea3bbdf37275210e082ce77cb010cec001b882af44c0ad0ee269f6661549f" Nov 04 05:01:50.081290 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:50.081215 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-19bf5b-7c6d784c4-s8rpn" Nov 04 05:01:50.133760 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:50.132926 4395 scope.go:122] "RemoveContainer" containerID="f4dea3bbdf37275210e082ce77cb010cec001b882af44c0ad0ee269f6661549f" Nov 04 05:01:50.133760 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:01:50.133370 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4dea3bbdf37275210e082ce77cb010cec001b882af44c0ad0ee269f6661549f\": container with ID starting with f4dea3bbdf37275210e082ce77cb010cec001b882af44c0ad0ee269f6661549f not found: ID does not exist" containerID="f4dea3bbdf37275210e082ce77cb010cec001b882af44c0ad0ee269f6661549f" Nov 04 05:01:50.133760 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:50.133420 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4dea3bbdf37275210e082ce77cb010cec001b882af44c0ad0ee269f6661549f"} err="failed to get container status \"f4dea3bbdf37275210e082ce77cb010cec001b882af44c0ad0ee269f6661549f\": rpc error: code = NotFound desc = could not find container \"f4dea3bbdf37275210e082ce77cb010cec001b882af44c0ad0ee269f6661549f\": container with ID starting with f4dea3bbdf37275210e082ce77cb010cec001b882af44c0ad0ee269f6661549f not found: ID does not exist" Nov 04 05:01:50.147323 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:50.147023 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-204e38/distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz" Nov 04 05:01:50.175321 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:50.174766 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-204e38/distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz" Nov 04 05:01:50.294323 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:50.283356 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-19bf5b-7c6d784c4-s8rpn"] Nov 04 05:01:50.298436 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:50.298401 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-19bf5b-7c6d784c4-s8rpn"] Nov 04 05:01:50.512276 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:50.512195 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5368507f-308f-447a-8da1-f6fc28c7a88d" path="/var/lib/kubelet/pods/5368507f-308f-447a-8da1-f6fc28c7a88d/volumes" Nov 04 05:01:51.126728 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:51.126690 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-204e38/distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz" Nov 04 05:01:55.548936 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:55.548896 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv"] Nov 04 05:01:55.549439 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:55.549130 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv" podUID="0574cb11-9f48-409a-8735-a4ab23411f4e" containerName="maas-api" containerID="cri-o://83b76a5920157704124c731b4eeb584253178feef18f18a916cb37ff025d9dbf" gracePeriod=30 Nov 04 05:01:55.836939 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:55.836916 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv" Nov 04 05:01:55.851143 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:55.851072 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0574cb11-9f48-409a-8735-a4ab23411f4e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0574cb11-9f48-409a-8735-a4ab23411f4e-openshift-service-ca\") pod \"0574cb11-9f48-409a-8735-a4ab23411f4e\" (UID: \"0574cb11-9f48-409a-8735-a4ab23411f4e\") " Nov 04 05:01:55.851279 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:55.851254 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0574cb11-9f48-409a-8735-a4ab23411f4e-kube-api-access-xpnt8\" (UniqueName: \"kubernetes.io/projected/0574cb11-9f48-409a-8735-a4ab23411f4e-kube-api-access-xpnt8\") pod \"0574cb11-9f48-409a-8735-a4ab23411f4e\" (UID: \"0574cb11-9f48-409a-8735-a4ab23411f4e\") " Nov 04 05:01:55.851371 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:55.851292 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0574cb11-9f48-409a-8735-a4ab23411f4e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0574cb11-9f48-409a-8735-a4ab23411f4e-maas-api-tls\") pod \"0574cb11-9f48-409a-8735-a4ab23411f4e\" (UID: \"0574cb11-9f48-409a-8735-a4ab23411f4e\") " Nov 04 05:01:55.851433 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:55.851392 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0574cb11-9f48-409a-8735-a4ab23411f4e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0574cb11-9f48-409a-8735-a4ab23411f4e-metrics-certs\") pod \"0574cb11-9f48-409a-8735-a4ab23411f4e\" (UID: \"0574cb11-9f48-409a-8735-a4ab23411f4e\") " Nov 04 05:01:55.851498 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:55.851470 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0574cb11-9f48-409a-8735-a4ab23411f4e-openshift-service-ca" pod "0574cb11-9f48-409a-8735-a4ab23411f4e" (UID: "0574cb11-9f48-409a-8735-a4ab23411f4e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:01:55.851759 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:55.851738 4395 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0574cb11-9f48-409a-8735-a4ab23411f4e-openshift-service-ca\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:55.853705 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:55.853676 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0574cb11-9f48-409a-8735-a4ab23411f4e-kube-api-access-xpnt8" pod "0574cb11-9f48-409a-8735-a4ab23411f4e" (UID: "0574cb11-9f48-409a-8735-a4ab23411f4e"). InnerVolumeSpecName "kube-api-access-xpnt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:01:55.853797 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:55.853696 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0574cb11-9f48-409a-8735-a4ab23411f4e-maas-api-tls" pod "0574cb11-9f48-409a-8735-a4ab23411f4e" (UID: "0574cb11-9f48-409a-8735-a4ab23411f4e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:01:55.853797 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:55.853677 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0574cb11-9f48-409a-8735-a4ab23411f4e-metrics-certs" pod "0574cb11-9f48-409a-8735-a4ab23411f4e" (UID: "0574cb11-9f48-409a-8735-a4ab23411f4e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:01:55.886051 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:55.886023 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11f35505-l69st"] Nov 04 05:01:55.886465 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:55.886451 4395 cpu_manager.go:414] "containerMap: removing container" podUID="24037ab8-fddf-4dd8-b170-96bdc4279e3f" containerName="payload-processing" Nov 04 05:01:55.886519 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:55.886467 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24037ab8-fddf-4dd8-b170-96bdc4279e3f" containerName="payload-processing" Nov 04 05:01:55.886519 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:55.886492 4395 cpu_manager.go:414] "containerMap: removing container" podUID="0574cb11-9f48-409a-8735-a4ab23411f4e" containerName="maas-api" Nov 04 05:01:55.886519 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:55.886498 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0574cb11-9f48-409a-8735-a4ab23411f4e" containerName="maas-api" Nov 04 05:01:55.886519 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:55.886510 4395 cpu_manager.go:414] "containerMap: removing container" podUID="5368507f-308f-447a-8da1-f6fc28c7a88d" containerName="payload-processing" Nov 04 05:01:55.886519 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:55.886515 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5368507f-308f-447a-8da1-f6fc28c7a88d" containerName="payload-processing" Nov 04 05:01:55.886674 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:55.886582 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="24037ab8-fddf-4dd8-b170-96bdc4279e3f" containerName="payload-processing" Nov 04 05:01:55.886674 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:55.886594 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="0574cb11-9f48-409a-8735-a4ab23411f4e" containerName="maas-api" Nov 04 05:01:55.886674 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:55.886600 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="5368507f-308f-447a-8da1-f6fc28c7a88d" containerName="payload-processing" Nov 04 05:01:55.890231 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:55.890210 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11f35505-l69st" Nov 04 05:01:55.938895 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:55.938865 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 05:01:55.939059 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:55.938899 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nz4vh\"" Nov 04 05:01:55.946382 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:55.946358 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11f35505-l69st"] Nov 04 05:01:55.952590 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:55.952563 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktrm4\" (UniqueName: \"kubernetes.io/projected/d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed-kube-api-access-ktrm4\") pod \"maas-api-revoke-sub-11f35505-l69st\" (UID: \"d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11f35505-l69st" Nov 04 05:01:55.952749 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:55.952608 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11f35505-l69st\" (UID: \"d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11f35505-l69st" Nov 04 05:01:55.952749 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:55.952718 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11f35505-l69st\" (UID: \"d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11f35505-l69st" Nov 04 05:01:55.952938 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:55.952918 4395 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0574cb11-9f48-409a-8735-a4ab23411f4e-metrics-certs\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:55.952999 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:55.952940 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpnt8\" (UniqueName: \"kubernetes.io/projected/0574cb11-9f48-409a-8735-a4ab23411f4e-kube-api-access-xpnt8\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:55.952999 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:55.952950 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0574cb11-9f48-409a-8735-a4ab23411f4e-maas-api-tls\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:56.053473 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:56.053431 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktrm4\" (UniqueName: \"kubernetes.io/projected/d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed-kube-api-access-ktrm4\") pod \"maas-api-revoke-sub-11f35505-l69st\" (UID: \"d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11f35505-l69st" Nov 04 05:01:56.053647 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:56.053560 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11f35505-l69st\" (UID: \"d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11f35505-l69st" Nov 04 05:01:56.053647 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:56.053606 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11f35505-l69st\" (UID: \"d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11f35505-l69st" Nov 04 05:01:56.054628 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:56.054599 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11f35505-l69st\" (UID: \"d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11f35505-l69st" Nov 04 05:01:56.056978 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:56.056953 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11f35505-l69st\" (UID: \"d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11f35505-l69st" Nov 04 05:01:56.111262 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:56.111187 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktrm4\" (UniqueName: \"kubernetes.io/projected/d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed-kube-api-access-ktrm4\") pod \"maas-api-revoke-sub-11f35505-l69st\" (UID: \"d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11f35505-l69st" Nov 04 05:01:56.129671 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:56.129635 4395 generic.go:356] "Generic (PLEG): container finished" podID="0574cb11-9f48-409a-8735-a4ab23411f4e" containerID="83b76a5920157704124c731b4eeb584253178feef18f18a916cb37ff025d9dbf" exitCode=0 Nov 04 05:01:56.129824 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:56.129686 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv" Nov 04 05:01:56.129824 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:56.129718 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv" event={"ID":"0574cb11-9f48-409a-8735-a4ab23411f4e","Type":"ContainerDied","Data":"83b76a5920157704124c731b4eeb584253178feef18f18a916cb37ff025d9dbf"} Nov 04 05:01:56.129824 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:56.129805 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv" event={"ID":"0574cb11-9f48-409a-8735-a4ab23411f4e","Type":"ContainerDied","Data":"b032d9b6beb1e3972c73a90cf5b0efcfb7d20f2d149fc6fb0aa2d8a24ffb95ae"} Nov 04 05:01:56.129990 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:56.129844 4395 scope.go:122] "RemoveContainer" containerID="83b76a5920157704124c731b4eeb584253178feef18f18a916cb37ff025d9dbf" Nov 04 05:01:56.140749 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:56.140732 4395 scope.go:122] "RemoveContainer" containerID="83b76a5920157704124c731b4eeb584253178feef18f18a916cb37ff025d9dbf" Nov 04 05:01:56.141040 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:01:56.141016 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83b76a5920157704124c731b4eeb584253178feef18f18a916cb37ff025d9dbf\": container with ID starting with 83b76a5920157704124c731b4eeb584253178feef18f18a916cb37ff025d9dbf not found: ID does not exist" containerID="83b76a5920157704124c731b4eeb584253178feef18f18a916cb37ff025d9dbf" Nov 04 05:01:56.141103 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:56.141051 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83b76a5920157704124c731b4eeb584253178feef18f18a916cb37ff025d9dbf"} err="failed to get container status \"83b76a5920157704124c731b4eeb584253178feef18f18a916cb37ff025d9dbf\": rpc error: code = NotFound desc = could not find container \"83b76a5920157704124c731b4eeb584253178feef18f18a916cb37ff025d9dbf\": container with ID starting with 83b76a5920157704124c731b4eeb584253178feef18f18a916cb37ff025d9dbf not found: ID does not exist" Nov 04 05:01:56.208760 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:56.208736 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11f35505-l69st" Nov 04 05:01:56.301216 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:56.301171 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv"] Nov 04 05:01:56.304945 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:56.304912 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv"] Nov 04 05:01:56.417271 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:56.417192 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0574cb11-9f48-409a-8735-a4ab23411f4e" path="/var/lib/kubelet/pods/0574cb11-9f48-409a-8735-a4ab23411f4e/volumes" Nov 04 05:01:56.593322 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:56.592139 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11f35505-l69st"] Nov 04 05:01:57.136916 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.136879 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11f35505-l69st" event={"ID":"d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed","Type":"ContainerStarted","Data":"863b224264c8290e82a49d7a719b15b2aecaf50ad9354491e04be67ad356da41"} Nov 04 05:01:57.137110 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.136969 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11f35505-l69st" event={"ID":"d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed","Type":"ContainerStarted","Data":"85f31cb1eda9fec3d0a550fa39734c5af3bd61cd67ecbcacb8407fdabdfa406f"} Nov 04 05:01:57.254170 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.254100 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11f35505-l69st" podStartSLOduration=2.254081003 podStartE2EDuration="2.254081003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:01:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:01:57.252326523 +0000 UTC m=+958.942408965" watchObservedRunningTime="2026-11-04 05:01:57.254081003 +0000 UTC m=+958.944163451" Nov 04 05:01:57.271017 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.270984 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985"] Nov 04 05:01:57.274834 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.274811 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985" Nov 04 05:01:57.323805 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.323779 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-19bf5b-openshift-default-dockercfg-5x94f\"" Nov 04 05:01:57.360782 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.360760 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985"] Nov 04 05:01:57.367350 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.367323 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-istio-token\") pod \"e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985\" (UID: \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\") " pod="openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985" Nov 04 05:01:57.367463 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.367379 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-workload-socket\") pod \"e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985\" (UID: \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\") " pod="openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985" Nov 04 05:01:57.367510 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.367458 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdmsk\" (UniqueName: \"kubernetes.io/projected/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-kube-api-access-zdmsk\") pod \"e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985\" (UID: \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\") " pod="openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985" Nov 04 05:01:57.367510 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.367495 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-workload-certs\") pod \"e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985\" (UID: \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\") " pod="openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985" Nov 04 05:01:57.367615 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.367598 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-istio-podinfo\") pod \"e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985\" (UID: \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\") " pod="openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985" Nov 04 05:01:57.367717 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.367700 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-credential-socket\") pod \"e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985\" (UID: \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\") " pod="openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985" Nov 04 05:01:57.367866 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.367736 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-istio-data\") pod \"e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985\" (UID: \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\") " pod="openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985" Nov 04 05:01:57.367866 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.367762 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-istiod-ca-cert\") pod \"e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985\" (UID: \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\") " pod="openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985" Nov 04 05:01:57.367866 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.367796 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-istio-envoy\") pod \"e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985\" (UID: \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\") " pod="openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985" Nov 04 05:01:57.468548 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.468497 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-istio-podinfo\") pod \"e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985\" (UID: \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\") " pod="openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985" Nov 04 05:01:57.468654 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.468613 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-credential-socket\") pod \"e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985\" (UID: \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\") " pod="openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985" Nov 04 05:01:57.468654 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.468647 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-istio-data\") pod \"e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985\" (UID: \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\") " pod="openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985" Nov 04 05:01:57.468773 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.468671 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-istiod-ca-cert\") pod \"e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985\" (UID: \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\") " pod="openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985" Nov 04 05:01:57.468773 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.468708 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-istio-envoy\") pod \"e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985\" (UID: \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\") " pod="openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985" Nov 04 05:01:57.468880 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.468807 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-istio-token\") pod \"e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985\" (UID: \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\") " pod="openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985" Nov 04 05:01:57.468936 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.468887 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-workload-socket\") pod \"e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985\" (UID: \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\") " pod="openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985" Nov 04 05:01:57.469170 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.468929 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdmsk\" (UniqueName: \"kubernetes.io/projected/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-kube-api-access-zdmsk\") pod \"e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985\" (UID: \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\") " pod="openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985" Nov 04 05:01:57.469170 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.468978 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-workload-certs\") pod \"e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985\" (UID: \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\") " pod="openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985" Nov 04 05:01:57.470113 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.469456 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-workload-certs\") pod \"e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985\" (UID: \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\") " pod="openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985" Nov 04 05:01:57.470113 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.469655 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-istio-data\") pod \"e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985\" (UID: \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\") " pod="openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985" Nov 04 05:01:57.470113 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.469771 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-workload-socket\") pod \"e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985\" (UID: \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\") " pod="openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985" Nov 04 05:01:57.470632 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.470611 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-istiod-ca-cert\") pod \"e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985\" (UID: \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\") " pod="openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985" Nov 04 05:01:57.470774 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.470741 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-credential-socket\") pod \"e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985\" (UID: \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\") " pod="openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985" Nov 04 05:01:57.471214 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.471195 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-istio-envoy\") pod \"e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985\" (UID: \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\") " pod="openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985" Nov 04 05:01:57.471692 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.471672 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-istio-podinfo\") pod \"e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985\" (UID: \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\") " pod="openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985" Nov 04 05:01:57.526651 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.526624 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-istio-token\") pod \"e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985\" (UID: \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\") " pod="openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985" Nov 04 05:01:57.526780 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.526762 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdmsk\" (UniqueName: \"kubernetes.io/projected/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-kube-api-access-zdmsk\") pod \"e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985\" (UID: \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\") " pod="openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985" Nov 04 05:01:57.576882 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.576858 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw"] Nov 04 05:01:57.584309 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.584277 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" Nov 04 05:01:57.588611 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.588588 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985" Nov 04 05:01:57.633043 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.633011 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-204fa0-openshift-default-dockercfg-h2wrr\"" Nov 04 05:01:57.643266 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.643242 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw"] Nov 04 05:01:57.670199 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.670170 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e36f89-3259-44b4-a0f3-d702c5774012-istio-envoy\") pod \"e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw\" (UID: \"b4e36f89-3259-44b4-a0f3-d702c5774012\") " pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" Nov 04 05:01:57.670332 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.670201 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e36f89-3259-44b4-a0f3-d702c5774012-istio-token\") pod \"e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw\" (UID: \"b4e36f89-3259-44b4-a0f3-d702c5774012\") " pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" Nov 04 05:01:57.670332 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.670224 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e36f89-3259-44b4-a0f3-d702c5774012-istio-podinfo\") pod \"e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw\" (UID: \"b4e36f89-3259-44b4-a0f3-d702c5774012\") " pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" Nov 04 05:01:57.670332 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.670266 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e36f89-3259-44b4-a0f3-d702c5774012-credential-socket\") pod \"e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw\" (UID: \"b4e36f89-3259-44b4-a0f3-d702c5774012\") " pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" Nov 04 05:01:57.670455 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.670416 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e36f89-3259-44b4-a0f3-d702c5774012-workload-socket\") pod \"e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw\" (UID: \"b4e36f89-3259-44b4-a0f3-d702c5774012\") " pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" Nov 04 05:01:57.670553 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.670537 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e36f89-3259-44b4-a0f3-d702c5774012-istiod-ca-cert\") pod \"e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw\" (UID: \"b4e36f89-3259-44b4-a0f3-d702c5774012\") " pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" Nov 04 05:01:57.670602 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.670587 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e36f89-3259-44b4-a0f3-d702c5774012-workload-certs\") pod \"e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw\" (UID: \"b4e36f89-3259-44b4-a0f3-d702c5774012\") " pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" Nov 04 05:01:57.670650 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.670617 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fvq5\" (UniqueName: \"kubernetes.io/projected/b4e36f89-3259-44b4-a0f3-d702c5774012-kube-api-access-6fvq5\") pod \"e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw\" (UID: \"b4e36f89-3259-44b4-a0f3-d702c5774012\") " pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" Nov 04 05:01:57.670741 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.670726 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e36f89-3259-44b4-a0f3-d702c5774012-istio-data\") pod \"e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw\" (UID: \"b4e36f89-3259-44b4-a0f3-d702c5774012\") " pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" Nov 04 05:01:57.772260 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.772160 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e36f89-3259-44b4-a0f3-d702c5774012-istiod-ca-cert\") pod \"e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw\" (UID: \"b4e36f89-3259-44b4-a0f3-d702c5774012\") " pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" Nov 04 05:01:57.773559 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.773501 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e36f89-3259-44b4-a0f3-d702c5774012-istiod-ca-cert\") pod \"e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw\" (UID: \"b4e36f89-3259-44b4-a0f3-d702c5774012\") " pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" Nov 04 05:01:57.774579 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.774556 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e36f89-3259-44b4-a0f3-d702c5774012-workload-certs\") pod \"e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw\" (UID: \"b4e36f89-3259-44b4-a0f3-d702c5774012\") " pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" Nov 04 05:01:57.775404 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.775372 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fvq5\" (UniqueName: \"kubernetes.io/projected/b4e36f89-3259-44b4-a0f3-d702c5774012-kube-api-access-6fvq5\") pod \"e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw\" (UID: \"b4e36f89-3259-44b4-a0f3-d702c5774012\") " pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" Nov 04 05:01:57.775515 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.775140 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e36f89-3259-44b4-a0f3-d702c5774012-workload-certs\") pod \"e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw\" (UID: \"b4e36f89-3259-44b4-a0f3-d702c5774012\") " pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" Nov 04 05:01:57.775515 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.775453 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e36f89-3259-44b4-a0f3-d702c5774012-istio-data\") pod \"e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw\" (UID: \"b4e36f89-3259-44b4-a0f3-d702c5774012\") " pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" Nov 04 05:01:57.775515 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.775493 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e36f89-3259-44b4-a0f3-d702c5774012-istio-envoy\") pod \"e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw\" (UID: \"b4e36f89-3259-44b4-a0f3-d702c5774012\") " pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" Nov 04 05:01:57.775515 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.775509 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e36f89-3259-44b4-a0f3-d702c5774012-istio-token\") pod \"e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw\" (UID: \"b4e36f89-3259-44b4-a0f3-d702c5774012\") " pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" Nov 04 05:01:57.775731 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.775532 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e36f89-3259-44b4-a0f3-d702c5774012-istio-podinfo\") pod \"e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw\" (UID: \"b4e36f89-3259-44b4-a0f3-d702c5774012\") " pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" Nov 04 05:01:57.775731 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.775558 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e36f89-3259-44b4-a0f3-d702c5774012-credential-socket\") pod \"e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw\" (UID: \"b4e36f89-3259-44b4-a0f3-d702c5774012\") " pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" Nov 04 05:01:57.775731 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.775614 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e36f89-3259-44b4-a0f3-d702c5774012-workload-socket\") pod \"e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw\" (UID: \"b4e36f89-3259-44b4-a0f3-d702c5774012\") " pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" Nov 04 05:01:57.775885 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.775864 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e36f89-3259-44b4-a0f3-d702c5774012-istio-data\") pod \"e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw\" (UID: \"b4e36f89-3259-44b4-a0f3-d702c5774012\") " pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" Nov 04 05:01:57.775957 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.775938 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e36f89-3259-44b4-a0f3-d702c5774012-workload-socket\") pod \"e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw\" (UID: \"b4e36f89-3259-44b4-a0f3-d702c5774012\") " pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" Nov 04 05:01:57.776254 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.776231 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e36f89-3259-44b4-a0f3-d702c5774012-credential-socket\") pod \"e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw\" (UID: \"b4e36f89-3259-44b4-a0f3-d702c5774012\") " pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" Nov 04 05:01:57.777866 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.777845 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e36f89-3259-44b4-a0f3-d702c5774012-istio-envoy\") pod \"e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw\" (UID: \"b4e36f89-3259-44b4-a0f3-d702c5774012\") " pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" Nov 04 05:01:57.778981 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.778958 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e36f89-3259-44b4-a0f3-d702c5774012-istio-podinfo\") pod \"e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw\" (UID: \"b4e36f89-3259-44b4-a0f3-d702c5774012\") " pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" Nov 04 05:01:57.781873 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.781847 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985"] Nov 04 05:01:57.834274 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.834241 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e36f89-3259-44b4-a0f3-d702c5774012-istio-token\") pod \"e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw\" (UID: \"b4e36f89-3259-44b4-a0f3-d702c5774012\") " pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" Nov 04 05:01:57.834543 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.834522 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fvq5\" (UniqueName: \"kubernetes.io/projected/b4e36f89-3259-44b4-a0f3-d702c5774012-kube-api-access-6fvq5\") pod \"e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw\" (UID: \"b4e36f89-3259-44b4-a0f3-d702c5774012\") " pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" Nov 04 05:01:57.900522 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:57.900500 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" Nov 04 05:01:58.021328 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:58.021277 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985"] Nov 04 05:01:58.024155 ip-10-0-134-224 kubenswrapper[4395]: W1104 05:01:58.024094 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1ba50cb_e1a3_421d_a71c_3d3c7dcd1983.slice/crio-e4df522a1d0b6d87ea8a13834a211f303ba72c0b37bd92c73a8ad75d447a14f0 WatchSource:0}: Error finding container e4df522a1d0b6d87ea8a13834a211f303ba72c0b37bd92c73a8ad75d447a14f0: Status 404 returned error can't find the container with id e4df522a1d0b6d87ea8a13834a211f303ba72c0b37bd92c73a8ad75d447a14f0 Nov 04 05:01:58.027039 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:58.027008 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 05:01:58.027123 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:58.027071 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 05:01:58.027123 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:58.027102 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 05:01:58.147015 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:58.146981 4395 generic.go:356] "Generic (PLEG): container finished" podID="d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed" containerID="863b224264c8290e82a49d7a719b15b2aecaf50ad9354491e04be67ad356da41" exitCode=0 Nov 04 05:01:58.147160 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:58.147017 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11f35505-l69st" event={"ID":"d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed","Type":"ContainerDied","Data":"863b224264c8290e82a49d7a719b15b2aecaf50ad9354491e04be67ad356da41"} Nov 04 05:01:58.148839 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:58.148817 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985" event={"ID":"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983","Type":"ContainerStarted","Data":"7565262b08628ca4e1b66dd7bb241a91d4562b03f70228b52f84dba62f050571"} Nov 04 05:01:58.148951 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:58.148855 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985" podUID="c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983" containerName="istio-proxy" containerID="cri-o://7565262b08628ca4e1b66dd7bb241a91d4562b03f70228b52f84dba62f050571" gracePeriod=30 Nov 04 05:01:58.148951 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:58.148863 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985" event={"ID":"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983","Type":"ContainerStarted","Data":"e4df522a1d0b6d87ea8a13834a211f303ba72c0b37bd92c73a8ad75d447a14f0"} Nov 04 05:01:58.312440 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:58.312402 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw"] Nov 04 05:01:58.313824 ip-10-0-134-224 kubenswrapper[4395]: W1104 05:01:58.313793 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e36f89_3259_44b4_a0f3_d702c5774012.slice/crio-2bea3ca514781b295ffcbd5b007f51b228358df53a8d55fffadf7eede01a1cf0 WatchSource:0}: Error finding container 2bea3ca514781b295ffcbd5b007f51b228358df53a8d55fffadf7eede01a1cf0: Status 404 returned error can't find the container with id 2bea3ca514781b295ffcbd5b007f51b228358df53a8d55fffadf7eede01a1cf0 Nov 04 05:01:58.316330 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:58.316253 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 05:01:58.316431 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:58.316343 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 05:01:58.316431 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:58.316376 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 05:01:58.373089 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:58.373043 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985" podStartSLOduration=1.373025066 podStartE2EDuration="1.373025066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:01:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:01:58.371479094 +0000 UTC m=+960.061561542" watchObservedRunningTime="2026-11-04 05:01:58.373025066 +0000 UTC m=+960.063107514" Nov 04 05:01:58.810098 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:58.810054 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-204fa0-679999f74-cpdhv" podUID="0574cb11-9f48-409a-8735-a4ab23411f4e" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.50:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 05:01:59.173319 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:59.172433 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" event={"ID":"b4e36f89-3259-44b4-a0f3-d702c5774012","Type":"ContainerStarted","Data":"30fd830947445a80fa2500d2f02edc5b3b4ce2328fa241ac61d1d77b7bfdd2b5"} Nov 04 05:01:59.173319 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:59.172529 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" event={"ID":"b4e36f89-3259-44b4-a0f3-d702c5774012","Type":"ContainerStarted","Data":"2bea3ca514781b295ffcbd5b007f51b228358df53a8d55fffadf7eede01a1cf0"} Nov 04 05:01:59.294093 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:59.294039 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" podStartSLOduration=2.294024559 podStartE2EDuration="2.294024559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:01:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:01:59.289889917 +0000 UTC m=+960.979972364" watchObservedRunningTime="2026-11-04 05:01:59.294024559 +0000 UTC m=+960.984107006" Nov 04 05:01:59.348061 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:59.348037 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11f35505-l69st" Nov 04 05:01:59.399903 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:59.399866 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed-maas-api-xks-ca\") pod \"d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed\" (UID: \"d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed\") " Nov 04 05:01:59.400073 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:59.399941 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed-kube-api-access-ktrm4\" (UniqueName: \"kubernetes.io/projected/d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed-kube-api-access-ktrm4\") pod \"d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed\" (UID: \"d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed\") " Nov 04 05:01:59.400073 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:59.399996 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed-maas-api-openshift-ca\") pod \"d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed\" (UID: \"d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed\") " Nov 04 05:01:59.400360 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:59.400331 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed-maas-api-openshift-ca" pod "d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed" (UID: "d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:01:59.400480 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:59.400413 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed-maas-api-openshift-ca\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:59.402087 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:59.402058 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed-maas-api-xks-ca" pod "d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed" (UID: "d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:01:59.402245 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:59.402226 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed-kube-api-access-ktrm4" pod "d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed" (UID: "d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed"). InnerVolumeSpecName "kube-api-access-ktrm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:01:59.502130 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:59.502054 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed-maas-api-xks-ca\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:59.502130 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:59.502082 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktrm4\" (UniqueName: \"kubernetes.io/projected/d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed-kube-api-access-ktrm4\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:01:59.900981 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:59.900950 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" Nov 04 05:01:59.902188 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:59.902155 4395 patch_prober.go:29] interesting pod/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 04 05:01:59.902337 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:01:59.902210 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" podUID="b4e36f89-3259-44b4-a0f3-d702c5774012" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 04 05:02:00.192378 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:00.192058 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11f35505-l69st" Nov 04 05:02:00.192378 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:00.192058 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11f35505-l69st" event={"ID":"d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed","Type":"ContainerDied","Data":"85f31cb1eda9fec3d0a550fa39734c5af3bd61cd67ecbcacb8407fdabdfa406f"} Nov 04 05:02:00.192378 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:00.192210 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85f31cb1eda9fec3d0a550fa39734c5af3bd61cd67ecbcacb8407fdabdfa406f" Nov 04 05:02:00.281319 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:00.280537 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c5bb6d/distinct-2-w2-c5bb6d-kserve-78dff78654-r524h"] Nov 04 05:02:00.902389 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:00.901713 4395 patch_prober.go:29] interesting pod/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 04 05:02:00.902389 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:00.901771 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" podUID="b4e36f89-3259-44b4-a0f3-d702c5774012" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 04 05:02:01.901231 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:01.901195 4395 patch_prober.go:29] interesting pod/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 04 05:02:01.901476 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:01.901250 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" podUID="b4e36f89-3259-44b4-a0f3-d702c5774012" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 04 05:02:02.900806 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:02.900765 4395 patch_prober.go:29] interesting pod/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 04 05:02:02.901284 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:02.900821 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" podUID="b4e36f89-3259-44b4-a0f3-d702c5774012" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 04 05:02:03.489110 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:02:03.488992 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 05:02:03.518147 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:03.518124 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985" Nov 04 05:02:03.552740 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:03.552706 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-workload-certs\") pod \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\" (UID: \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\") " Nov 04 05:02:03.552833 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:03.552763 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-istio-token\" (UniqueName: \"kubernetes.io/projected/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-istio-token\") pod \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\" (UID: \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\") " Nov 04 05:02:03.552833 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:03.552785 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-kube-api-access-zdmsk\" (UniqueName: \"kubernetes.io/projected/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-kube-api-access-zdmsk\") pod \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\" (UID: \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\") " Nov 04 05:02:03.552833 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:03.552813 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-istio-data\") pod \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\" (UID: \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\") " Nov 04 05:02:03.552978 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:03.552838 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-credential-socket\") pod \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\" (UID: \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\") " Nov 04 05:02:03.552978 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:03.552944 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-workload-certs" pod "c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983" (UID: "c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:03.552978 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:03.552948 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-istio-envoy\") pod \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\" (UID: \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\") " Nov 04 05:02:03.553123 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:03.553027 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-istio-podinfo\") pod \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\" (UID: \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\") " Nov 04 05:02:03.553176 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:03.553131 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-credential-socket" pod "c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983" (UID: "c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:03.553176 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:03.553143 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-istio-data" pod "c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983" (UID: "c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:03.553176 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:03.553169 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-workload-socket\") pod \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\" (UID: \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\") " Nov 04 05:02:03.553368 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:03.553206 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-istiod-ca-cert\") pod \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\" (UID: \"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983\") " Nov 04 05:02:03.553463 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:03.553441 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-workload-socket" pod "c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983" (UID: "c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:03.553638 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:03.553607 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-istiod-ca-cert" pod "c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983" (UID: "c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:02:03.553728 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:03.553675 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-workload-certs\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:03.553728 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:03.553694 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-istio-data\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:03.553728 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:03.553709 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-credential-socket\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:03.553728 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:03.553723 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-workload-socket\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:03.555093 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:03.555049 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-istio-envoy" pod "c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983" (UID: "c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:03.555182 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:03.555130 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-kube-api-access-zdmsk" pod "c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983" (UID: "c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983"). InnerVolumeSpecName "kube-api-access-zdmsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:02:03.555234 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:03.555211 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-istio-token" pod "c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983" (UID: "c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:02:03.555272 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:03.555214 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-istio-podinfo" pod "c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983" (UID: "c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:02:03.654536 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:03.654499 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-istio-token\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:03.654536 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:03.654531 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdmsk\" (UniqueName: \"kubernetes.io/projected/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-kube-api-access-zdmsk\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:03.654824 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:03.654546 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-istio-envoy\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:03.654824 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:03.654559 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-istio-podinfo\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:03.654824 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:03.654571 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983-istiod-ca-cert\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:03.901237 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:03.901200 4395 patch_prober.go:29] interesting pod/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 04 05:02:03.901677 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:03.901265 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" podUID="b4e36f89-3259-44b4-a0f3-d702c5774012" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 04 05:02:04.126515 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:04.126479 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw"] Nov 04 05:02:04.126679 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:04.126664 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" podUID="b4e36f89-3259-44b4-a0f3-d702c5774012" containerName="istio-proxy" containerID="cri-o://30fd830947445a80fa2500d2f02edc5b3b4ce2328fa241ac61d1d77b7bfdd2b5" gracePeriod=30 Nov 04 05:02:04.235397 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:04.235014 4395 generic.go:356] "Generic (PLEG): container finished" podID="c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983" containerID="7565262b08628ca4e1b66dd7bb241a91d4562b03f70228b52f84dba62f050571" exitCode=0 Nov 04 05:02:04.235397 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:04.235072 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985" Nov 04 05:02:04.235397 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:04.235153 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985" event={"ID":"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983","Type":"ContainerDied","Data":"7565262b08628ca4e1b66dd7bb241a91d4562b03f70228b52f84dba62f050571"} Nov 04 05:02:04.235397 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:04.235194 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985" event={"ID":"c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983","Type":"ContainerDied","Data":"e4df522a1d0b6d87ea8a13834a211f303ba72c0b37bd92c73a8ad75d447a14f0"} Nov 04 05:02:04.235397 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:04.235219 4395 scope.go:122] "RemoveContainer" containerID="7565262b08628ca4e1b66dd7bb241a91d4562b03f70228b52f84dba62f050571" Nov 04 05:02:04.248218 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:04.248196 4395 scope.go:122] "RemoveContainer" containerID="7565262b08628ca4e1b66dd7bb241a91d4562b03f70228b52f84dba62f050571" Nov 04 05:02:04.248559 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:02:04.248535 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7565262b08628ca4e1b66dd7bb241a91d4562b03f70228b52f84dba62f050571\": container with ID starting with 7565262b08628ca4e1b66dd7bb241a91d4562b03f70228b52f84dba62f050571 not found: ID does not exist" containerID="7565262b08628ca4e1b66dd7bb241a91d4562b03f70228b52f84dba62f050571" Nov 04 05:02:04.248685 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:04.248569 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7565262b08628ca4e1b66dd7bb241a91d4562b03f70228b52f84dba62f050571"} err="failed to get container status \"7565262b08628ca4e1b66dd7bb241a91d4562b03f70228b52f84dba62f050571\": rpc error: code = NotFound desc = could not find container \"7565262b08628ca4e1b66dd7bb241a91d4562b03f70228b52f84dba62f050571\": container with ID starting with 7565262b08628ca4e1b66dd7bb241a91d4562b03f70228b52f84dba62f050571 not found: ID does not exist" Nov 04 05:02:04.434352 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:04.431822 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985"] Nov 04 05:02:04.434352 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:04.431876 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-19bf5b-openshift-default-5c54fdccbf-th985"] Nov 04 05:02:06.415258 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:06.415227 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983" path="/var/lib/kubelet/pods/c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983/volumes" Nov 04 05:02:07.022604 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:07.022569 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl"] Nov 04 05:02:07.253230 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:07.253103 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" podUID="518e3ddc-9b6e-4127-a47c-397d9117a3ae" containerName="istio-proxy" containerID="cri-o://f9f218bbf8a0b763a2ac2ef92fbb5325d8afbbe69a706e4b94d6c61c3a7cb527" gracePeriod=30 Nov 04 05:02:08.560789 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:08.560748 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468"] Nov 04 05:02:08.561284 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:08.561017 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" podUID="24ea2807-ab22-495c-b66a-b35111e36b97" containerName="istio-proxy" containerID="cri-o://4eb04862a55327c22d4d40edd6f67fcb5868355224af7e0d5663fe70bba37ddf" gracePeriod=30 Nov 04 05:02:09.171833 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:09.171803 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-a48f7e/distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p"] Nov 04 05:02:09.172368 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:09.172350 4395 cpu_manager.go:414] "containerMap: removing container" podUID="d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed" containerName="revoke-keys" Nov 04 05:02:09.172479 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:09.172369 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed" containerName="revoke-keys" Nov 04 05:02:09.172479 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:09.172400 4395 cpu_manager.go:414] "containerMap: removing container" podUID="c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983" containerName="istio-proxy" Nov 04 05:02:09.172479 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:09.172409 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983" containerName="istio-proxy" Nov 04 05:02:09.172633 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:09.172512 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed" containerName="revoke-keys" Nov 04 05:02:09.172633 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:09.172528 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1ba50cb-e1a3-421d-a71c-3d3c7dcd1983" containerName="istio-proxy" Nov 04 05:02:09.175874 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:09.175853 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a48f7e/distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p" Nov 04 05:02:09.224615 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:09.224595 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-a48f7e\"/\"distinct-2-w4-a48f7e-kserve-self-signed-certs\"" Nov 04 05:02:09.233685 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:09.233663 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-a48f7e/distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p"] Nov 04 05:02:09.322586 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:09.322550 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/124ecb9d-bcbe-415a-9ba7-5789af9afb98-tls-certs\") pod \"distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p\" (UID: \"124ecb9d-bcbe-415a-9ba7-5789af9afb98\") " pod="e2e-models-e2e-worker-w4-a48f7e/distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p" Nov 04 05:02:09.322748 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:09.322726 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124ecb9d-bcbe-415a-9ba7-5789af9afb98-model-cache\") pod \"distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p\" (UID: \"124ecb9d-bcbe-415a-9ba7-5789af9afb98\") " pod="e2e-models-e2e-worker-w4-a48f7e/distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p" Nov 04 05:02:09.322826 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:09.322793 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbjn2\" (UniqueName: \"kubernetes.io/projected/124ecb9d-bcbe-415a-9ba7-5789af9afb98-kube-api-access-pbjn2\") pod \"distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p\" (UID: \"124ecb9d-bcbe-415a-9ba7-5789af9afb98\") " pod="e2e-models-e2e-worker-w4-a48f7e/distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p" Nov 04 05:02:09.323020 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:09.322895 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124ecb9d-bcbe-415a-9ba7-5789af9afb98-tmp-dir\") pod \"distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p\" (UID: \"124ecb9d-bcbe-415a-9ba7-5789af9afb98\") " pod="e2e-models-e2e-worker-w4-a48f7e/distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p" Nov 04 05:02:09.323020 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:09.322992 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/124ecb9d-bcbe-415a-9ba7-5789af9afb98-dshm\") pod \"distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p\" (UID: \"124ecb9d-bcbe-415a-9ba7-5789af9afb98\") " pod="e2e-models-e2e-worker-w4-a48f7e/distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p" Nov 04 05:02:09.323148 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:09.323021 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/124ecb9d-bcbe-415a-9ba7-5789af9afb98-home\") pod \"distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p\" (UID: \"124ecb9d-bcbe-415a-9ba7-5789af9afb98\") " pod="e2e-models-e2e-worker-w4-a48f7e/distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p" Nov 04 05:02:09.424566 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:09.424481 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbjn2\" (UniqueName: \"kubernetes.io/projected/124ecb9d-bcbe-415a-9ba7-5789af9afb98-kube-api-access-pbjn2\") pod \"distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p\" (UID: \"124ecb9d-bcbe-415a-9ba7-5789af9afb98\") " pod="e2e-models-e2e-worker-w4-a48f7e/distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p" Nov 04 05:02:09.424566 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:09.424530 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124ecb9d-bcbe-415a-9ba7-5789af9afb98-tmp-dir\") pod \"distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p\" (UID: \"124ecb9d-bcbe-415a-9ba7-5789af9afb98\") " pod="e2e-models-e2e-worker-w4-a48f7e/distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p" Nov 04 05:02:09.424790 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:09.424612 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/124ecb9d-bcbe-415a-9ba7-5789af9afb98-dshm\") pod \"distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p\" (UID: \"124ecb9d-bcbe-415a-9ba7-5789af9afb98\") " pod="e2e-models-e2e-worker-w4-a48f7e/distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p" Nov 04 05:02:09.424790 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:09.424636 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/124ecb9d-bcbe-415a-9ba7-5789af9afb98-home\") pod \"distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p\" (UID: \"124ecb9d-bcbe-415a-9ba7-5789af9afb98\") " pod="e2e-models-e2e-worker-w4-a48f7e/distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p" Nov 04 05:02:09.424790 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:09.424720 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/124ecb9d-bcbe-415a-9ba7-5789af9afb98-tls-certs\") pod \"distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p\" (UID: \"124ecb9d-bcbe-415a-9ba7-5789af9afb98\") " pod="e2e-models-e2e-worker-w4-a48f7e/distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p" Nov 04 05:02:09.424790 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:09.424776 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124ecb9d-bcbe-415a-9ba7-5789af9afb98-model-cache\") pod \"distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p\" (UID: \"124ecb9d-bcbe-415a-9ba7-5789af9afb98\") " pod="e2e-models-e2e-worker-w4-a48f7e/distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p" Nov 04 05:02:09.424989 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:09.424967 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124ecb9d-bcbe-415a-9ba7-5789af9afb98-tmp-dir\") pod \"distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p\" (UID: \"124ecb9d-bcbe-415a-9ba7-5789af9afb98\") " pod="e2e-models-e2e-worker-w4-a48f7e/distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p" Nov 04 05:02:09.425076 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:09.425054 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/124ecb9d-bcbe-415a-9ba7-5789af9afb98-home\") pod \"distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p\" (UID: \"124ecb9d-bcbe-415a-9ba7-5789af9afb98\") " pod="e2e-models-e2e-worker-w4-a48f7e/distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p" Nov 04 05:02:09.425129 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:09.425100 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124ecb9d-bcbe-415a-9ba7-5789af9afb98-model-cache\") pod \"distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p\" (UID: \"124ecb9d-bcbe-415a-9ba7-5789af9afb98\") " pod="e2e-models-e2e-worker-w4-a48f7e/distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p" Nov 04 05:02:09.426907 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:09.426881 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/124ecb9d-bcbe-415a-9ba7-5789af9afb98-dshm\") pod \"distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p\" (UID: \"124ecb9d-bcbe-415a-9ba7-5789af9afb98\") " pod="e2e-models-e2e-worker-w4-a48f7e/distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p" Nov 04 05:02:09.427133 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:09.427114 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/124ecb9d-bcbe-415a-9ba7-5789af9afb98-tls-certs\") pod \"distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p\" (UID: \"124ecb9d-bcbe-415a-9ba7-5789af9afb98\") " pod="e2e-models-e2e-worker-w4-a48f7e/distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p" Nov 04 05:02:09.481820 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:09.481785 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbjn2\" (UniqueName: \"kubernetes.io/projected/124ecb9d-bcbe-415a-9ba7-5789af9afb98-kube-api-access-pbjn2\") pod \"distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p\" (UID: \"124ecb9d-bcbe-415a-9ba7-5789af9afb98\") " pod="e2e-models-e2e-worker-w4-a48f7e/distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p" Nov 04 05:02:09.487463 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:09.487436 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a48f7e/distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p" Nov 04 05:02:09.871293 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:09.871269 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-a48f7e/distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p"] Nov 04 05:02:09.872632 ip-10-0-134-224 kubenswrapper[4395]: W1104 05:02:09.872599 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod124ecb9d_bcbe_415a_9ba7_5789af9afb98.slice/crio-2e7025449bef8e4a0348af47844d7b366af6c20c0556834d3465528bb03ec81f WatchSource:0}: Error finding container 2e7025449bef8e4a0348af47844d7b366af6c20c0556834d3465528bb03ec81f: Status 404 returned error can't find the container with id 2e7025449bef8e4a0348af47844d7b366af6c20c0556834d3465528bb03ec81f Nov 04 05:02:10.271372 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:10.271340 4395 generic.go:356] "Generic (PLEG): container finished" podID="b4e36f89-3259-44b4-a0f3-d702c5774012" containerID="30fd830947445a80fa2500d2f02edc5b3b4ce2328fa241ac61d1d77b7bfdd2b5" exitCode=0 Nov 04 05:02:10.271489 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:10.271416 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" event={"ID":"b4e36f89-3259-44b4-a0f3-d702c5774012","Type":"ContainerDied","Data":"30fd830947445a80fa2500d2f02edc5b3b4ce2328fa241ac61d1d77b7bfdd2b5"} Nov 04 05:02:10.273645 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:10.273602 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a48f7e/distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p" event={"ID":"124ecb9d-bcbe-415a-9ba7-5789af9afb98","Type":"ContainerStarted","Data":"b4e6be4e2be9679058cb9cdf79fd824472db7f46011fac4a9b09e98eb7c583a9"} Nov 04 05:02:10.273777 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:10.273660 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a48f7e/distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p" event={"ID":"124ecb9d-bcbe-415a-9ba7-5789af9afb98","Type":"ContainerStarted","Data":"2e7025449bef8e4a0348af47844d7b366af6c20c0556834d3465528bb03ec81f"} Nov 04 05:02:10.368026 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:10.368004 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" Nov 04 05:02:10.383922 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:10.383871 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-a48f7e/distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p" podStartSLOduration=1.383858118 podStartE2EDuration="1.383858118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:02:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:02:10.38193431 +0000 UTC m=+972.072016757" watchObservedRunningTime="2026-11-04 05:02:10.383858118 +0000 UTC m=+972.073940564" Nov 04 05:02:10.435186 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:10.435078 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e36f89-3259-44b4-a0f3-d702c5774012-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e36f89-3259-44b4-a0f3-d702c5774012-istio-data\") pod \"b4e36f89-3259-44b4-a0f3-d702c5774012\" (UID: \"b4e36f89-3259-44b4-a0f3-d702c5774012\") " Nov 04 05:02:10.435186 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:10.435115 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e36f89-3259-44b4-a0f3-d702c5774012-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e36f89-3259-44b4-a0f3-d702c5774012-workload-socket\") pod \"b4e36f89-3259-44b4-a0f3-d702c5774012\" (UID: \"b4e36f89-3259-44b4-a0f3-d702c5774012\") " Nov 04 05:02:10.435186 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:10.435140 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e36f89-3259-44b4-a0f3-d702c5774012-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e36f89-3259-44b4-a0f3-d702c5774012-istio-envoy\") pod \"b4e36f89-3259-44b4-a0f3-d702c5774012\" (UID: \"b4e36f89-3259-44b4-a0f3-d702c5774012\") " Nov 04 05:02:10.435186 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:10.435185 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4e36f89-3259-44b4-a0f3-d702c5774012-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e36f89-3259-44b4-a0f3-d702c5774012-istiod-ca-cert\") pod \"b4e36f89-3259-44b4-a0f3-d702c5774012\" (UID: \"b4e36f89-3259-44b4-a0f3-d702c5774012\") " Nov 04 05:02:10.435513 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:10.435224 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4e36f89-3259-44b4-a0f3-d702c5774012-istio-token\" (UniqueName: \"kubernetes.io/projected/b4e36f89-3259-44b4-a0f3-d702c5774012-istio-token\") pod \"b4e36f89-3259-44b4-a0f3-d702c5774012\" (UID: \"b4e36f89-3259-44b4-a0f3-d702c5774012\") " Nov 04 05:02:10.435513 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:10.435291 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4e36f89-3259-44b4-a0f3-d702c5774012-kube-api-access-6fvq5\" (UniqueName: \"kubernetes.io/projected/b4e36f89-3259-44b4-a0f3-d702c5774012-kube-api-access-6fvq5\") pod \"b4e36f89-3259-44b4-a0f3-d702c5774012\" (UID: \"b4e36f89-3259-44b4-a0f3-d702c5774012\") " Nov 04 05:02:10.435513 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:10.435354 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4e36f89-3259-44b4-a0f3-d702c5774012-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e36f89-3259-44b4-a0f3-d702c5774012-istio-podinfo\") pod \"b4e36f89-3259-44b4-a0f3-d702c5774012\" (UID: \"b4e36f89-3259-44b4-a0f3-d702c5774012\") " Nov 04 05:02:10.435513 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:10.435365 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e36f89-3259-44b4-a0f3-d702c5774012-workload-socket" pod "b4e36f89-3259-44b4-a0f3-d702c5774012" (UID: "b4e36f89-3259-44b4-a0f3-d702c5774012"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:10.435513 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:10.435379 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e36f89-3259-44b4-a0f3-d702c5774012-istio-data" pod "b4e36f89-3259-44b4-a0f3-d702c5774012" (UID: "b4e36f89-3259-44b4-a0f3-d702c5774012"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:10.435513 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:10.435384 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e36f89-3259-44b4-a0f3-d702c5774012-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e36f89-3259-44b4-a0f3-d702c5774012-credential-socket\") pod \"b4e36f89-3259-44b4-a0f3-d702c5774012\" (UID: \"b4e36f89-3259-44b4-a0f3-d702c5774012\") " Nov 04 05:02:10.435513 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:10.435490 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e36f89-3259-44b4-a0f3-d702c5774012-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e36f89-3259-44b4-a0f3-d702c5774012-workload-certs\") pod \"b4e36f89-3259-44b4-a0f3-d702c5774012\" (UID: \"b4e36f89-3259-44b4-a0f3-d702c5774012\") " Nov 04 05:02:10.436065 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:10.435606 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4e36f89-3259-44b4-a0f3-d702c5774012-istiod-ca-cert" pod "b4e36f89-3259-44b4-a0f3-d702c5774012" (UID: "b4e36f89-3259-44b4-a0f3-d702c5774012"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:02:10.436065 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:10.435746 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e36f89-3259-44b4-a0f3-d702c5774012-workload-certs" pod "b4e36f89-3259-44b4-a0f3-d702c5774012" (UID: "b4e36f89-3259-44b4-a0f3-d702c5774012"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:10.436065 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:10.435923 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e36f89-3259-44b4-a0f3-d702c5774012-istiod-ca-cert\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:10.436065 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:10.435941 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e36f89-3259-44b4-a0f3-d702c5774012-workload-certs\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:10.436065 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:10.435956 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e36f89-3259-44b4-a0f3-d702c5774012-istio-data\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:10.436065 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:10.435970 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e36f89-3259-44b4-a0f3-d702c5774012-workload-socket\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:10.436490 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:10.436173 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e36f89-3259-44b4-a0f3-d702c5774012-credential-socket" pod "b4e36f89-3259-44b4-a0f3-d702c5774012" (UID: "b4e36f89-3259-44b4-a0f3-d702c5774012"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:10.437688 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:10.437653 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e36f89-3259-44b4-a0f3-d702c5774012-istio-envoy" pod "b4e36f89-3259-44b4-a0f3-d702c5774012" (UID: "b4e36f89-3259-44b4-a0f3-d702c5774012"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:10.437920 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:10.437896 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e36f89-3259-44b4-a0f3-d702c5774012-istio-token" pod "b4e36f89-3259-44b4-a0f3-d702c5774012" (UID: "b4e36f89-3259-44b4-a0f3-d702c5774012"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:02:10.438265 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:10.438248 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4e36f89-3259-44b4-a0f3-d702c5774012-istio-podinfo" pod "b4e36f89-3259-44b4-a0f3-d702c5774012" (UID: "b4e36f89-3259-44b4-a0f3-d702c5774012"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:02:10.438684 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:10.438669 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e36f89-3259-44b4-a0f3-d702c5774012-kube-api-access-6fvq5" pod "b4e36f89-3259-44b4-a0f3-d702c5774012" (UID: "b4e36f89-3259-44b4-a0f3-d702c5774012"). InnerVolumeSpecName "kube-api-access-6fvq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:02:10.487672 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:10.487649 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a48f7e/distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p" Nov 04 05:02:10.537332 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:10.537287 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e36f89-3259-44b4-a0f3-d702c5774012-istio-envoy\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:10.537480 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:10.537349 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e36f89-3259-44b4-a0f3-d702c5774012-istio-token\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:10.537480 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:10.537361 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fvq5\" (UniqueName: \"kubernetes.io/projected/b4e36f89-3259-44b4-a0f3-d702c5774012-kube-api-access-6fvq5\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:10.537480 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:10.537372 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e36f89-3259-44b4-a0f3-d702c5774012-istio-podinfo\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:10.537480 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:10.537381 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e36f89-3259-44b4-a0f3-d702c5774012-credential-socket\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:11.280743 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:11.280715 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" Nov 04 05:02:11.280743 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:11.280724 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw" event={"ID":"b4e36f89-3259-44b4-a0f3-d702c5774012","Type":"ContainerDied","Data":"2bea3ca514781b295ffcbd5b007f51b228358df53a8d55fffadf7eede01a1cf0"} Nov 04 05:02:11.281320 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:11.280776 4395 scope.go:122] "RemoveContainer" containerID="30fd830947445a80fa2500d2f02edc5b3b4ce2328fa241ac61d1d77b7bfdd2b5" Nov 04 05:02:11.481186 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:11.481166 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw"] Nov 04 05:02:11.485479 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:11.485455 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-204fa0-openshift-default-5cb9795595-gcvkw"] Nov 04 05:02:11.740621 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:11.740594 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5"] Nov 04 05:02:11.740992 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:11.740980 4395 cpu_manager.go:414] "containerMap: removing container" podUID="b4e36f89-3259-44b4-a0f3-d702c5774012" containerName="istio-proxy" Nov 04 05:02:11.741031 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:11.740993 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4e36f89-3259-44b4-a0f3-d702c5774012" containerName="istio-proxy" Nov 04 05:02:11.741085 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:11.741077 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4e36f89-3259-44b4-a0f3-d702c5774012" containerName="istio-proxy" Nov 04 05:02:11.746066 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:11.746048 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" Nov 04 05:02:11.808503 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:11.808483 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5"] Nov 04 05:02:11.847917 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:11.847893 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3d7a106-096e-4551-819f-c8b166554914-istio-data\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5\" (UID: \"a3d7a106-096e-4551-819f-c8b166554914\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" Nov 04 05:02:11.848030 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:11.847941 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3d7a106-096e-4551-819f-c8b166554914-istiod-ca-cert\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5\" (UID: \"a3d7a106-096e-4551-819f-c8b166554914\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" Nov 04 05:02:11.848078 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:11.848035 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3d7a106-096e-4551-819f-c8b166554914-workload-certs\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5\" (UID: \"a3d7a106-096e-4551-819f-c8b166554914\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" Nov 04 05:02:11.848078 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:11.848065 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3d7a106-096e-4551-819f-c8b166554914-istio-envoy\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5\" (UID: \"a3d7a106-096e-4551-819f-c8b166554914\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" Nov 04 05:02:11.848172 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:11.848091 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d7a106-096e-4551-819f-c8b166554914-workload-socket\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5\" (UID: \"a3d7a106-096e-4551-819f-c8b166554914\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" Nov 04 05:02:11.848209 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:11.848183 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3d7a106-096e-4551-819f-c8b166554914-istio-token\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5\" (UID: \"a3d7a106-096e-4551-819f-c8b166554914\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" Nov 04 05:02:11.848250 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:11.848232 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3d7a106-096e-4551-819f-c8b166554914-istio-podinfo\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5\" (UID: \"a3d7a106-096e-4551-819f-c8b166554914\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" Nov 04 05:02:11.848372 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:11.848355 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d7a106-096e-4551-819f-c8b166554914-credential-socket\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5\" (UID: \"a3d7a106-096e-4551-819f-c8b166554914\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" Nov 04 05:02:11.848449 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:11.848413 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttgsv\" (UniqueName: \"kubernetes.io/projected/a3d7a106-096e-4551-819f-c8b166554914-kube-api-access-ttgsv\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5\" (UID: \"a3d7a106-096e-4551-819f-c8b166554914\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" Nov 04 05:02:11.949714 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:11.949689 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3d7a106-096e-4551-819f-c8b166554914-istio-data\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5\" (UID: \"a3d7a106-096e-4551-819f-c8b166554914\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" Nov 04 05:02:11.949829 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:11.949729 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3d7a106-096e-4551-819f-c8b166554914-istiod-ca-cert\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5\" (UID: \"a3d7a106-096e-4551-819f-c8b166554914\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" Nov 04 05:02:11.949829 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:11.949764 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3d7a106-096e-4551-819f-c8b166554914-workload-certs\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5\" (UID: \"a3d7a106-096e-4551-819f-c8b166554914\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" Nov 04 05:02:11.949829 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:11.949788 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3d7a106-096e-4551-819f-c8b166554914-istio-envoy\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5\" (UID: \"a3d7a106-096e-4551-819f-c8b166554914\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" Nov 04 05:02:11.949829 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:11.949821 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d7a106-096e-4551-819f-c8b166554914-workload-socket\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5\" (UID: \"a3d7a106-096e-4551-819f-c8b166554914\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" Nov 04 05:02:11.950078 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:11.949850 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3d7a106-096e-4551-819f-c8b166554914-istio-token\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5\" (UID: \"a3d7a106-096e-4551-819f-c8b166554914\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" Nov 04 05:02:11.950078 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:11.949906 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3d7a106-096e-4551-819f-c8b166554914-istio-podinfo\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5\" (UID: \"a3d7a106-096e-4551-819f-c8b166554914\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" Nov 04 05:02:11.950078 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:11.949952 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d7a106-096e-4551-819f-c8b166554914-credential-socket\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5\" (UID: \"a3d7a106-096e-4551-819f-c8b166554914\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" Nov 04 05:02:11.950078 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:11.950007 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttgsv\" (UniqueName: \"kubernetes.io/projected/a3d7a106-096e-4551-819f-c8b166554914-kube-api-access-ttgsv\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5\" (UID: \"a3d7a106-096e-4551-819f-c8b166554914\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" Nov 04 05:02:11.950334 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:11.950107 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3d7a106-096e-4551-819f-c8b166554914-istio-data\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5\" (UID: \"a3d7a106-096e-4551-819f-c8b166554914\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" Nov 04 05:02:11.950334 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:11.950319 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3d7a106-096e-4551-819f-c8b166554914-workload-certs\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5\" (UID: \"a3d7a106-096e-4551-819f-c8b166554914\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" Nov 04 05:02:11.950428 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:11.950335 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d7a106-096e-4551-819f-c8b166554914-workload-socket\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5\" (UID: \"a3d7a106-096e-4551-819f-c8b166554914\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" Nov 04 05:02:11.950428 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:11.950399 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d7a106-096e-4551-819f-c8b166554914-credential-socket\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5\" (UID: \"a3d7a106-096e-4551-819f-c8b166554914\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" Nov 04 05:02:11.950558 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:11.950542 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3d7a106-096e-4551-819f-c8b166554914-istiod-ca-cert\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5\" (UID: \"a3d7a106-096e-4551-819f-c8b166554914\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" Nov 04 05:02:11.952118 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:11.952091 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3d7a106-096e-4551-819f-c8b166554914-istio-envoy\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5\" (UID: \"a3d7a106-096e-4551-819f-c8b166554914\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" Nov 04 05:02:11.952245 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:11.952224 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3d7a106-096e-4551-819f-c8b166554914-istio-podinfo\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5\" (UID: \"a3d7a106-096e-4551-819f-c8b166554914\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" Nov 04 05:02:12.025436 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.025389 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3d7a106-096e-4551-819f-c8b166554914-istio-token\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5\" (UID: \"a3d7a106-096e-4551-819f-c8b166554914\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" Nov 04 05:02:12.033130 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.033111 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttgsv\" (UniqueName: \"kubernetes.io/projected/a3d7a106-096e-4551-819f-c8b166554914-kube-api-access-ttgsv\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5\" (UID: \"a3d7a106-096e-4551-819f-c8b166554914\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" Nov 04 05:02:12.060039 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.060021 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" Nov 04 05:02:12.419973 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.419924 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4e36f89-3259-44b4-a0f3-d702c5774012" path="/var/lib/kubelet/pods/b4e36f89-3259-44b4-a0f3-d702c5774012/volumes" Nov 04 05:02:12.490444 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.490336 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5"] Nov 04 05:02:12.491399 ip-10-0-134-224 kubenswrapper[4395]: W1104 05:02:12.491375 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3d7a106_096e_4551_819f_c8b166554914.slice/crio-9250642e5642d7a0a2b79172e04c5beada54c53e3f6273888d95bad9eb184463 WatchSource:0}: Error finding container 9250642e5642d7a0a2b79172e04c5beada54c53e3f6273888d95bad9eb184463: Status 404 returned error can't find the container with id 9250642e5642d7a0a2b79172e04c5beada54c53e3f6273888d95bad9eb184463 Nov 04 05:02:12.493885 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.493844 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 05:02:12.493970 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.493923 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 05:02:12.493970 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.493957 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 05:02:12.505238 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.505212 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" Nov 04 05:02:12.658617 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.658529 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/518e3ddc-9b6e-4127-a47c-397d9117a3ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/518e3ddc-9b6e-4127-a47c-397d9117a3ae-istiod-ca-cert\") pod \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\" (UID: \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\") " Nov 04 05:02:12.658783 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.658641 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/518e3ddc-9b6e-4127-a47c-397d9117a3ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/518e3ddc-9b6e-4127-a47c-397d9117a3ae-credential-socket\") pod \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\" (UID: \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\") " Nov 04 05:02:12.658783 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.658713 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/518e3ddc-9b6e-4127-a47c-397d9117a3ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/518e3ddc-9b6e-4127-a47c-397d9117a3ae-istio-data\") pod \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\" (UID: \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\") " Nov 04 05:02:12.658783 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.658776 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/518e3ddc-9b6e-4127-a47c-397d9117a3ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/518e3ddc-9b6e-4127-a47c-397d9117a3ae-istio-envoy\") pod \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\" (UID: \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\") " Nov 04 05:02:12.658955 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.658806 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/518e3ddc-9b6e-4127-a47c-397d9117a3ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/518e3ddc-9b6e-4127-a47c-397d9117a3ae-istio-podinfo\") pod \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\" (UID: \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\") " Nov 04 05:02:12.658955 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.658843 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/518e3ddc-9b6e-4127-a47c-397d9117a3ae-istio-token\" (UniqueName: \"kubernetes.io/projected/518e3ddc-9b6e-4127-a47c-397d9117a3ae-istio-token\") pod \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\" (UID: \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\") " Nov 04 05:02:12.658955 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.658870 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/518e3ddc-9b6e-4127-a47c-397d9117a3ae-kube-api-access-bn7z9\" (UniqueName: \"kubernetes.io/projected/518e3ddc-9b6e-4127-a47c-397d9117a3ae-kube-api-access-bn7z9\") pod \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\" (UID: \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\") " Nov 04 05:02:12.658955 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.658908 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/518e3ddc-9b6e-4127-a47c-397d9117a3ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/518e3ddc-9b6e-4127-a47c-397d9117a3ae-workload-socket\") pod \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\" (UID: \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\") " Nov 04 05:02:12.658955 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.658934 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/518e3ddc-9b6e-4127-a47c-397d9117a3ae-istiod-ca-cert" pod "518e3ddc-9b6e-4127-a47c-397d9117a3ae" (UID: "518e3ddc-9b6e-4127-a47c-397d9117a3ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:02:12.659221 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.658957 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/518e3ddc-9b6e-4127-a47c-397d9117a3ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/518e3ddc-9b6e-4127-a47c-397d9117a3ae-workload-certs\") pod \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\" (UID: \"518e3ddc-9b6e-4127-a47c-397d9117a3ae\") " Nov 04 05:02:12.659221 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.658985 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/518e3ddc-9b6e-4127-a47c-397d9117a3ae-credential-socket" pod "518e3ddc-9b6e-4127-a47c-397d9117a3ae" (UID: "518e3ddc-9b6e-4127-a47c-397d9117a3ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:12.659403 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.659382 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/518e3ddc-9b6e-4127-a47c-397d9117a3ae-istiod-ca-cert\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:12.659482 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.659408 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/518e3ddc-9b6e-4127-a47c-397d9117a3ae-credential-socket\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:12.659535 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.659488 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/518e3ddc-9b6e-4127-a47c-397d9117a3ae-istio-data" pod "518e3ddc-9b6e-4127-a47c-397d9117a3ae" (UID: "518e3ddc-9b6e-4127-a47c-397d9117a3ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:12.659868 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.659813 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/518e3ddc-9b6e-4127-a47c-397d9117a3ae-workload-socket" pod "518e3ddc-9b6e-4127-a47c-397d9117a3ae" (UID: "518e3ddc-9b6e-4127-a47c-397d9117a3ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:12.660010 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.659910 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/518e3ddc-9b6e-4127-a47c-397d9117a3ae-workload-certs" pod "518e3ddc-9b6e-4127-a47c-397d9117a3ae" (UID: "518e3ddc-9b6e-4127-a47c-397d9117a3ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:12.662115 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.662089 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/518e3ddc-9b6e-4127-a47c-397d9117a3ae-istio-token" pod "518e3ddc-9b6e-4127-a47c-397d9117a3ae" (UID: "518e3ddc-9b6e-4127-a47c-397d9117a3ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:02:12.665613 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.665587 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/518e3ddc-9b6e-4127-a47c-397d9117a3ae-istio-podinfo" pod "518e3ddc-9b6e-4127-a47c-397d9117a3ae" (UID: "518e3ddc-9b6e-4127-a47c-397d9117a3ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:02:12.665613 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.665605 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/518e3ddc-9b6e-4127-a47c-397d9117a3ae-kube-api-access-bn7z9" pod "518e3ddc-9b6e-4127-a47c-397d9117a3ae" (UID: "518e3ddc-9b6e-4127-a47c-397d9117a3ae"). InnerVolumeSpecName "kube-api-access-bn7z9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:02:12.666747 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.666726 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/518e3ddc-9b6e-4127-a47c-397d9117a3ae-istio-envoy" pod "518e3ddc-9b6e-4127-a47c-397d9117a3ae" (UID: "518e3ddc-9b6e-4127-a47c-397d9117a3ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:12.699304 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.699275 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-157af145-tt25h"] Nov 04 05:02:12.699907 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.699888 4395 cpu_manager.go:414] "containerMap: removing container" podUID="518e3ddc-9b6e-4127-a47c-397d9117a3ae" containerName="istio-proxy" Nov 04 05:02:12.699907 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.699906 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="518e3ddc-9b6e-4127-a47c-397d9117a3ae" containerName="istio-proxy" Nov 04 05:02:12.700071 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.700013 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="518e3ddc-9b6e-4127-a47c-397d9117a3ae" containerName="istio-proxy" Nov 04 05:02:12.704571 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.704555 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157af145-tt25h" Nov 04 05:02:12.753484 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.753460 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 05:02:12.753592 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.753556 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 05:02:12.753592 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.753555 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 05:02:12.753661 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.753619 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nz4vh\"" Nov 04 05:02:12.760560 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.760534 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/518e3ddc-9b6e-4127-a47c-397d9117a3ae-workload-certs\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:12.760794 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.760769 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/518e3ddc-9b6e-4127-a47c-397d9117a3ae-istio-data\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:12.760794 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.760796 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/518e3ddc-9b6e-4127-a47c-397d9117a3ae-istio-envoy\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:12.760999 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.760810 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/518e3ddc-9b6e-4127-a47c-397d9117a3ae-istio-podinfo\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:12.760999 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.760824 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/518e3ddc-9b6e-4127-a47c-397d9117a3ae-istio-token\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:12.760999 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.760838 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn7z9\" (UniqueName: \"kubernetes.io/projected/518e3ddc-9b6e-4127-a47c-397d9117a3ae-kube-api-access-bn7z9\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:12.760999 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.760854 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/518e3ddc-9b6e-4127-a47c-397d9117a3ae-workload-socket\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:12.762049 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.762024 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-157af145-tt25h"] Nov 04 05:02:12.861894 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.861868 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/906b0195-e1ec-4930-89b1-3b4fb3c1bbdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-157af145-tt25h\" (UID: \"906b0195-e1ec-4930-89b1-3b4fb3c1bbdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157af145-tt25h" Nov 04 05:02:12.862065 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.861916 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/906b0195-e1ec-4930-89b1-3b4fb3c1bbdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-157af145-tt25h\" (UID: \"906b0195-e1ec-4930-89b1-3b4fb3c1bbdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157af145-tt25h" Nov 04 05:02:12.862065 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.861997 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw24c\" (UniqueName: \"kubernetes.io/projected/906b0195-e1ec-4930-89b1-3b4fb3c1bbdf-kube-api-access-xw24c\") pod \"maas-api-revoke-sub-157af145-tt25h\" (UID: \"906b0195-e1ec-4930-89b1-3b4fb3c1bbdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157af145-tt25h" Nov 04 05:02:12.965398 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.964442 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/906b0195-e1ec-4930-89b1-3b4fb3c1bbdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-157af145-tt25h\" (UID: \"906b0195-e1ec-4930-89b1-3b4fb3c1bbdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157af145-tt25h" Nov 04 05:02:12.965398 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.964535 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/906b0195-e1ec-4930-89b1-3b4fb3c1bbdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-157af145-tt25h\" (UID: \"906b0195-e1ec-4930-89b1-3b4fb3c1bbdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157af145-tt25h" Nov 04 05:02:12.965398 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.964598 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw24c\" (UniqueName: \"kubernetes.io/projected/906b0195-e1ec-4930-89b1-3b4fb3c1bbdf-kube-api-access-xw24c\") pod \"maas-api-revoke-sub-157af145-tt25h\" (UID: \"906b0195-e1ec-4930-89b1-3b4fb3c1bbdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157af145-tt25h" Nov 04 05:02:12.972580 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.972258 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/906b0195-e1ec-4930-89b1-3b4fb3c1bbdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-157af145-tt25h\" (UID: \"906b0195-e1ec-4930-89b1-3b4fb3c1bbdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157af145-tt25h" Nov 04 05:02:12.975424 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:12.975391 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/906b0195-e1ec-4930-89b1-3b4fb3c1bbdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-157af145-tt25h\" (UID: \"906b0195-e1ec-4930-89b1-3b4fb3c1bbdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157af145-tt25h" Nov 04 05:02:13.026334 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:13.022268 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw24c\" (UniqueName: \"kubernetes.io/projected/906b0195-e1ec-4930-89b1-3b4fb3c1bbdf-kube-api-access-xw24c\") pod \"maas-api-revoke-sub-157af145-tt25h\" (UID: \"906b0195-e1ec-4930-89b1-3b4fb3c1bbdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157af145-tt25h" Nov 04 05:02:13.299772 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:13.299736 4395 generic.go:356] "Generic (PLEG): container finished" podID="518e3ddc-9b6e-4127-a47c-397d9117a3ae" containerID="f9f218bbf8a0b763a2ac2ef92fbb5325d8afbbe69a706e4b94d6c61c3a7cb527" exitCode=0 Nov 04 05:02:13.299935 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:13.299799 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" Nov 04 05:02:13.299935 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:13.299816 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" event={"ID":"518e3ddc-9b6e-4127-a47c-397d9117a3ae","Type":"ContainerDied","Data":"f9f218bbf8a0b763a2ac2ef92fbb5325d8afbbe69a706e4b94d6c61c3a7cb527"} Nov 04 05:02:13.299935 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:13.299893 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl" event={"ID":"518e3ddc-9b6e-4127-a47c-397d9117a3ae","Type":"ContainerDied","Data":"b186a0d838fc5addb967d5882ad892985e9f21880c400272863334404e133964"} Nov 04 05:02:13.299935 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:13.299911 4395 scope.go:122] "RemoveContainer" containerID="f9f218bbf8a0b763a2ac2ef92fbb5325d8afbbe69a706e4b94d6c61c3a7cb527" Nov 04 05:02:13.301819 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:13.301795 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" event={"ID":"a3d7a106-096e-4551-819f-c8b166554914","Type":"ContainerStarted","Data":"2bbbefde8fe524095f1c456d56cdb7dbc0abb470b208f02d14cea8700557cfac"} Nov 04 05:02:13.301940 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:13.301849 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" event={"ID":"a3d7a106-096e-4551-819f-c8b166554914","Type":"ContainerStarted","Data":"9250642e5642d7a0a2b79172e04c5beada54c53e3f6273888d95bad9eb184463"} Nov 04 05:02:13.310191 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:13.310173 4395 scope.go:122] "RemoveContainer" containerID="43742d519f2ebe3a7ef9a14ded98694cd286e629d0ed8159c3b8a4adf0cc81d4" Nov 04 05:02:13.319066 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:13.319049 4395 scope.go:122] "RemoveContainer" containerID="f9f218bbf8a0b763a2ac2ef92fbb5325d8afbbe69a706e4b94d6c61c3a7cb527" Nov 04 05:02:13.319345 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:02:13.319321 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9f218bbf8a0b763a2ac2ef92fbb5325d8afbbe69a706e4b94d6c61c3a7cb527\": container with ID starting with f9f218bbf8a0b763a2ac2ef92fbb5325d8afbbe69a706e4b94d6c61c3a7cb527 not found: ID does not exist" containerID="f9f218bbf8a0b763a2ac2ef92fbb5325d8afbbe69a706e4b94d6c61c3a7cb527" Nov 04 05:02:13.319440 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:13.319351 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9f218bbf8a0b763a2ac2ef92fbb5325d8afbbe69a706e4b94d6c61c3a7cb527"} err="failed to get container status \"f9f218bbf8a0b763a2ac2ef92fbb5325d8afbbe69a706e4b94d6c61c3a7cb527\": rpc error: code = NotFound desc = could not find container \"f9f218bbf8a0b763a2ac2ef92fbb5325d8afbbe69a706e4b94d6c61c3a7cb527\": container with ID starting with f9f218bbf8a0b763a2ac2ef92fbb5325d8afbbe69a706e4b94d6c61c3a7cb527 not found: ID does not exist" Nov 04 05:02:13.319440 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:13.319368 4395 scope.go:122] "RemoveContainer" containerID="43742d519f2ebe3a7ef9a14ded98694cd286e629d0ed8159c3b8a4adf0cc81d4" Nov 04 05:02:13.319607 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:02:13.319590 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43742d519f2ebe3a7ef9a14ded98694cd286e629d0ed8159c3b8a4adf0cc81d4\": container with ID starting with 43742d519f2ebe3a7ef9a14ded98694cd286e629d0ed8159c3b8a4adf0cc81d4 not found: ID does not exist" containerID="43742d519f2ebe3a7ef9a14ded98694cd286e629d0ed8159c3b8a4adf0cc81d4" Nov 04 05:02:13.319651 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:13.319613 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43742d519f2ebe3a7ef9a14ded98694cd286e629d0ed8159c3b8a4adf0cc81d4"} err="failed to get container status \"43742d519f2ebe3a7ef9a14ded98694cd286e629d0ed8159c3b8a4adf0cc81d4\": rpc error: code = NotFound desc = could not find container \"43742d519f2ebe3a7ef9a14ded98694cd286e629d0ed8159c3b8a4adf0cc81d4\": container with ID starting with 43742d519f2ebe3a7ef9a14ded98694cd286e629d0ed8159c3b8a4adf0cc81d4 not found: ID does not exist" Nov 04 05:02:13.320729 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:13.320712 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157af145-tt25h" Nov 04 05:02:13.485518 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:13.485455 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" podStartSLOduration=2.485443797 podStartE2EDuration="2.485443797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:02:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:02:13.48480759 +0000 UTC m=+975.174890045" watchObservedRunningTime="2026-11-04 05:02:13.485443797 +0000 UTC m=+975.175526244" Nov 04 05:02:13.591457 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:13.591425 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl"] Nov 04 05:02:13.602709 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:13.596511 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e62689-openshift-default-b5d66459-rbjfl"] Nov 04 05:02:13.729138 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:13.729106 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-157af145-tt25h"] Nov 04 05:02:13.729697 ip-10-0-134-224 kubenswrapper[4395]: W1104 05:02:13.729670 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod906b0195_e1ec_4930_89b1_3b4fb3c1bbdf.slice/crio-a416fc62b98f2ed1e3ceae0c6902b71f94b3e4fb4d135c007d063955e7c83320 WatchSource:0}: Error finding container a416fc62b98f2ed1e3ceae0c6902b71f94b3e4fb4d135c007d063955e7c83320: Status 404 returned error can't find the container with id a416fc62b98f2ed1e3ceae0c6902b71f94b3e4fb4d135c007d063955e7c83320 Nov 04 05:02:14.061786 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:14.061757 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" Nov 04 05:02:14.062125 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:14.062103 4395 patch_prober.go:29] interesting pod/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 04 05:02:14.062293 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:14.062272 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" podUID="a3d7a106-096e-4551-819f-c8b166554914" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 04 05:02:14.171497 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:14.171152 4395 patch_prober.go:29] interesting pod/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 04 05:02:14.171497 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:14.171210 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" podUID="24ea2807-ab22-495c-b66a-b35111e36b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 04 05:02:14.349794 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:14.349754 4395 generic.go:356] "Generic (PLEG): container finished" podID="24ea2807-ab22-495c-b66a-b35111e36b97" containerID="4eb04862a55327c22d4d40edd6f67fcb5868355224af7e0d5663fe70bba37ddf" exitCode=0 Nov 04 05:02:14.350034 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:14.350017 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" event={"ID":"24ea2807-ab22-495c-b66a-b35111e36b97","Type":"ContainerDied","Data":"4eb04862a55327c22d4d40edd6f67fcb5868355224af7e0d5663fe70bba37ddf"} Nov 04 05:02:14.350204 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:14.350191 4395 scope.go:122] "RemoveContainer" containerID="fb40a3de103e8fda221f07cce71ea4956f46b331866584e14b44b2409a8dc3d0" Nov 04 05:02:14.359631 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:14.359607 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157af145-tt25h" event={"ID":"906b0195-e1ec-4930-89b1-3b4fb3c1bbdf","Type":"ContainerStarted","Data":"a416fc62b98f2ed1e3ceae0c6902b71f94b3e4fb4d135c007d063955e7c83320"} Nov 04 05:02:14.447256 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:14.421777 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="518e3ddc-9b6e-4127-a47c-397d9117a3ae" path="/var/lib/kubelet/pods/518e3ddc-9b6e-4127-a47c-397d9117a3ae/volumes" Nov 04 05:02:14.606340 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:14.606268 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" Nov 04 05:02:14.703737 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:14.703704 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24ea2807-ab22-495c-b66a-b35111e36b97-kube-api-access-w7ph4\" (UniqueName: \"kubernetes.io/projected/24ea2807-ab22-495c-b66a-b35111e36b97-kube-api-access-w7ph4\") pod \"24ea2807-ab22-495c-b66a-b35111e36b97\" (UID: \"24ea2807-ab22-495c-b66a-b35111e36b97\") " Nov 04 05:02:14.703737 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:14.703741 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24ea2807-ab22-495c-b66a-b35111e36b97-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24ea2807-ab22-495c-b66a-b35111e36b97-istio-podinfo\") pod \"24ea2807-ab22-495c-b66a-b35111e36b97\" (UID: \"24ea2807-ab22-495c-b66a-b35111e36b97\") " Nov 04 05:02:14.703967 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:14.703761 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24ea2807-ab22-495c-b66a-b35111e36b97-istio-token\" (UniqueName: \"kubernetes.io/projected/24ea2807-ab22-495c-b66a-b35111e36b97-istio-token\") pod \"24ea2807-ab22-495c-b66a-b35111e36b97\" (UID: \"24ea2807-ab22-495c-b66a-b35111e36b97\") " Nov 04 05:02:14.703967 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:14.703805 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24ea2807-ab22-495c-b66a-b35111e36b97-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24ea2807-ab22-495c-b66a-b35111e36b97-workload-socket\") pod \"24ea2807-ab22-495c-b66a-b35111e36b97\" (UID: \"24ea2807-ab22-495c-b66a-b35111e36b97\") " Nov 04 05:02:14.703967 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:14.703826 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24ea2807-ab22-495c-b66a-b35111e36b97-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24ea2807-ab22-495c-b66a-b35111e36b97-istio-data\") pod \"24ea2807-ab22-495c-b66a-b35111e36b97\" (UID: \"24ea2807-ab22-495c-b66a-b35111e36b97\") " Nov 04 05:02:14.703967 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:14.703847 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24ea2807-ab22-495c-b66a-b35111e36b97-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24ea2807-ab22-495c-b66a-b35111e36b97-credential-socket\") pod \"24ea2807-ab22-495c-b66a-b35111e36b97\" (UID: \"24ea2807-ab22-495c-b66a-b35111e36b97\") " Nov 04 05:02:14.703967 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:14.703888 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24ea2807-ab22-495c-b66a-b35111e36b97-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24ea2807-ab22-495c-b66a-b35111e36b97-istio-envoy\") pod \"24ea2807-ab22-495c-b66a-b35111e36b97\" (UID: \"24ea2807-ab22-495c-b66a-b35111e36b97\") " Nov 04 05:02:14.703967 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:14.703929 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24ea2807-ab22-495c-b66a-b35111e36b97-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24ea2807-ab22-495c-b66a-b35111e36b97-istiod-ca-cert\") pod \"24ea2807-ab22-495c-b66a-b35111e36b97\" (UID: \"24ea2807-ab22-495c-b66a-b35111e36b97\") " Nov 04 05:02:14.703967 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:14.703967 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24ea2807-ab22-495c-b66a-b35111e36b97-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24ea2807-ab22-495c-b66a-b35111e36b97-workload-certs\") pod \"24ea2807-ab22-495c-b66a-b35111e36b97\" (UID: \"24ea2807-ab22-495c-b66a-b35111e36b97\") " Nov 04 05:02:14.704359 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:14.704139 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ea2807-ab22-495c-b66a-b35111e36b97-credential-socket" pod "24ea2807-ab22-495c-b66a-b35111e36b97" (UID: "24ea2807-ab22-495c-b66a-b35111e36b97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:14.704359 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:14.704178 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ea2807-ab22-495c-b66a-b35111e36b97-istio-data" pod "24ea2807-ab22-495c-b66a-b35111e36b97" (UID: "24ea2807-ab22-495c-b66a-b35111e36b97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:14.704476 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:14.704425 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ea2807-ab22-495c-b66a-b35111e36b97-workload-certs" pod "24ea2807-ab22-495c-b66a-b35111e36b97" (UID: "24ea2807-ab22-495c-b66a-b35111e36b97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:14.704476 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:14.704438 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24ea2807-ab22-495c-b66a-b35111e36b97-istiod-ca-cert" pod "24ea2807-ab22-495c-b66a-b35111e36b97" (UID: "24ea2807-ab22-495c-b66a-b35111e36b97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:02:14.704476 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:14.704457 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ea2807-ab22-495c-b66a-b35111e36b97-workload-socket" pod "24ea2807-ab22-495c-b66a-b35111e36b97" (UID: "24ea2807-ab22-495c-b66a-b35111e36b97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:14.704619 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:14.704503 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24ea2807-ab22-495c-b66a-b35111e36b97-istio-data\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:14.704619 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:14.704523 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24ea2807-ab22-495c-b66a-b35111e36b97-credential-socket\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:14.704619 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:14.704538 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24ea2807-ab22-495c-b66a-b35111e36b97-workload-certs\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:14.706175 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:14.706140 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ea2807-ab22-495c-b66a-b35111e36b97-istio-envoy" pod "24ea2807-ab22-495c-b66a-b35111e36b97" (UID: "24ea2807-ab22-495c-b66a-b35111e36b97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:14.706350 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:14.706226 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24ea2807-ab22-495c-b66a-b35111e36b97-istio-podinfo" pod "24ea2807-ab22-495c-b66a-b35111e36b97" (UID: "24ea2807-ab22-495c-b66a-b35111e36b97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:02:14.706458 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:14.706434 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ea2807-ab22-495c-b66a-b35111e36b97-kube-api-access-w7ph4" pod "24ea2807-ab22-495c-b66a-b35111e36b97" (UID: "24ea2807-ab22-495c-b66a-b35111e36b97"). InnerVolumeSpecName "kube-api-access-w7ph4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:02:14.706533 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:14.706493 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ea2807-ab22-495c-b66a-b35111e36b97-istio-token" pod "24ea2807-ab22-495c-b66a-b35111e36b97" (UID: "24ea2807-ab22-495c-b66a-b35111e36b97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:02:14.805405 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:14.805375 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24ea2807-ab22-495c-b66a-b35111e36b97-istiod-ca-cert\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:14.805405 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:14.805401 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7ph4\" (UniqueName: \"kubernetes.io/projected/24ea2807-ab22-495c-b66a-b35111e36b97-kube-api-access-w7ph4\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:14.805405 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:14.805412 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24ea2807-ab22-495c-b66a-b35111e36b97-istio-podinfo\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:14.805689 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:14.805421 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24ea2807-ab22-495c-b66a-b35111e36b97-istio-token\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:14.805689 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:14.805431 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24ea2807-ab22-495c-b66a-b35111e36b97-workload-socket\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:14.805689 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:14.805441 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24ea2807-ab22-495c-b66a-b35111e36b97-istio-envoy\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:15.080323 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:15.077495 4395 patch_prober.go:29] interesting pod/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 04 05:02:15.080323 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:15.077558 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" podUID="a3d7a106-096e-4551-819f-c8b166554914" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 04 05:02:15.312155 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:15.312101 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-c5bb6d/distinct-2-w2-c5bb6d-kserve-78dff78654-r524h" podUID="8f554177-f75c-438e-8d57-71fdd73f7d2e" containerName="main" containerID="cri-o://4bb152bc105e6af21a730417de47d0cba974262ca855b8dfe99f7e3ed51bf453" gracePeriod=45 Nov 04 05:02:15.389676 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:15.389648 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" Nov 04 05:02:15.389841 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:15.389647 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468" event={"ID":"24ea2807-ab22-495c-b66a-b35111e36b97","Type":"ContainerDied","Data":"06f7d5e6df70e090521670fa4b2aded7c92398d7277da075b5e41408ca570225"} Nov 04 05:02:15.389841 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:15.389808 4395 scope.go:122] "RemoveContainer" containerID="4eb04862a55327c22d4d40edd6f67fcb5868355224af7e0d5663fe70bba37ddf" Nov 04 05:02:15.391781 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:15.391758 4395 generic.go:356] "Generic (PLEG): container finished" podID="906b0195-e1ec-4930-89b1-3b4fb3c1bbdf" containerID="d1b4e6b9192a02e43f0ced2592c115caac59f5e4eb59f63016c445e70ed13a1b" exitCode=0 Nov 04 05:02:15.391916 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:15.391843 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157af145-tt25h" event={"ID":"906b0195-e1ec-4930-89b1-3b4fb3c1bbdf","Type":"ContainerDied","Data":"d1b4e6b9192a02e43f0ced2592c115caac59f5e4eb59f63016c445e70ed13a1b"} Nov 04 05:02:15.618395 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:15.618363 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c5bb6d/distinct-2-w2-c5bb6d-kserve-78dff78654-r524h" Nov 04 05:02:15.656157 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:15.656131 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468"] Nov 04 05:02:15.684147 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:15.684112 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-db8fae-openshift-default-55bbb8f84f-5n468"] Nov 04 05:02:15.714631 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:15.714610 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f554177-f75c-438e-8d57-71fdd73f7d2e-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f554177-f75c-438e-8d57-71fdd73f7d2e-tls-certs\") pod \"8f554177-f75c-438e-8d57-71fdd73f7d2e\" (UID: \"8f554177-f75c-438e-8d57-71fdd73f7d2e\") " Nov 04 05:02:15.714736 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:15.714665 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f554177-f75c-438e-8d57-71fdd73f7d2e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f554177-f75c-438e-8d57-71fdd73f7d2e-dshm\") pod \"8f554177-f75c-438e-8d57-71fdd73f7d2e\" (UID: \"8f554177-f75c-438e-8d57-71fdd73f7d2e\") " Nov 04 05:02:15.714736 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:15.714728 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f554177-f75c-438e-8d57-71fdd73f7d2e-kube-api-access-lv4mj\" (UniqueName: \"kubernetes.io/projected/8f554177-f75c-438e-8d57-71fdd73f7d2e-kube-api-access-lv4mj\") pod \"8f554177-f75c-438e-8d57-71fdd73f7d2e\" (UID: \"8f554177-f75c-438e-8d57-71fdd73f7d2e\") " Nov 04 05:02:15.714814 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:15.714801 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f554177-f75c-438e-8d57-71fdd73f7d2e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f554177-f75c-438e-8d57-71fdd73f7d2e-model-cache\") pod \"8f554177-f75c-438e-8d57-71fdd73f7d2e\" (UID: \"8f554177-f75c-438e-8d57-71fdd73f7d2e\") " Nov 04 05:02:15.714850 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:15.714827 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f554177-f75c-438e-8d57-71fdd73f7d2e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f554177-f75c-438e-8d57-71fdd73f7d2e-tmp-dir\") pod \"8f554177-f75c-438e-8d57-71fdd73f7d2e\" (UID: \"8f554177-f75c-438e-8d57-71fdd73f7d2e\") " Nov 04 05:02:15.714850 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:15.714845 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f554177-f75c-438e-8d57-71fdd73f7d2e-home\" (UniqueName: \"kubernetes.io/empty-dir/8f554177-f75c-438e-8d57-71fdd73f7d2e-home\") pod \"8f554177-f75c-438e-8d57-71fdd73f7d2e\" (UID: \"8f554177-f75c-438e-8d57-71fdd73f7d2e\") " Nov 04 05:02:15.715060 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:15.715017 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f554177-f75c-438e-8d57-71fdd73f7d2e-model-cache" pod "8f554177-f75c-438e-8d57-71fdd73f7d2e" (UID: "8f554177-f75c-438e-8d57-71fdd73f7d2e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:15.715181 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:15.715070 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f554177-f75c-438e-8d57-71fdd73f7d2e-tmp-dir" pod "8f554177-f75c-438e-8d57-71fdd73f7d2e" (UID: "8f554177-f75c-438e-8d57-71fdd73f7d2e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:15.715238 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:15.715188 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f554177-f75c-438e-8d57-71fdd73f7d2e-home" pod "8f554177-f75c-438e-8d57-71fdd73f7d2e" (UID: "8f554177-f75c-438e-8d57-71fdd73f7d2e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:15.717000 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:15.716977 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f554177-f75c-438e-8d57-71fdd73f7d2e-dshm" pod "8f554177-f75c-438e-8d57-71fdd73f7d2e" (UID: "8f554177-f75c-438e-8d57-71fdd73f7d2e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:15.717084 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:15.717041 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f554177-f75c-438e-8d57-71fdd73f7d2e-tls-certs" pod "8f554177-f75c-438e-8d57-71fdd73f7d2e" (UID: "8f554177-f75c-438e-8d57-71fdd73f7d2e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:02:15.717084 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:15.717045 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f554177-f75c-438e-8d57-71fdd73f7d2e-kube-api-access-lv4mj" pod "8f554177-f75c-438e-8d57-71fdd73f7d2e" (UID: "8f554177-f75c-438e-8d57-71fdd73f7d2e"). InnerVolumeSpecName "kube-api-access-lv4mj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:02:15.816392 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:15.816361 4395 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f554177-f75c-438e-8d57-71fdd73f7d2e-model-cache\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:15.816392 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:15.816394 4395 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f554177-f75c-438e-8d57-71fdd73f7d2e-tmp-dir\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:15.816560 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:15.816405 4395 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f554177-f75c-438e-8d57-71fdd73f7d2e-home\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:15.816560 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:15.816412 4395 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f554177-f75c-438e-8d57-71fdd73f7d2e-tls-certs\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:15.816560 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:15.816420 4395 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f554177-f75c-438e-8d57-71fdd73f7d2e-dshm\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:15.816560 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:15.816430 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv4mj\" (UniqueName: \"kubernetes.io/projected/8f554177-f75c-438e-8d57-71fdd73f7d2e-kube-api-access-lv4mj\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:16.076174 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:16.061677 4395 patch_prober.go:29] interesting pod/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 04 05:02:16.076174 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:16.061737 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" podUID="a3d7a106-096e-4551-819f-c8b166554914" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 04 05:02:16.415595 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:16.415562 4395 generic.go:356] "Generic (PLEG): container finished" podID="8f554177-f75c-438e-8d57-71fdd73f7d2e" containerID="4bb152bc105e6af21a730417de47d0cba974262ca855b8dfe99f7e3ed51bf453" exitCode=0 Nov 04 05:02:16.415788 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:16.415775 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c5bb6d/distinct-2-w2-c5bb6d-kserve-78dff78654-r524h" Nov 04 05:02:16.420196 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:16.420167 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24ea2807-ab22-495c-b66a-b35111e36b97" path="/var/lib/kubelet/pods/24ea2807-ab22-495c-b66a-b35111e36b97/volumes" Nov 04 05:02:16.420604 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:16.420585 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c5bb6d/distinct-2-w2-c5bb6d-kserve-78dff78654-r524h" event={"ID":"8f554177-f75c-438e-8d57-71fdd73f7d2e","Type":"ContainerDied","Data":"4bb152bc105e6af21a730417de47d0cba974262ca855b8dfe99f7e3ed51bf453"} Nov 04 05:02:16.420671 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:16.420616 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c5bb6d/distinct-2-w2-c5bb6d-kserve-78dff78654-r524h" event={"ID":"8f554177-f75c-438e-8d57-71fdd73f7d2e","Type":"ContainerDied","Data":"2d9279d7a2b0514f7f0db8f384b7c7810947ce5d050166029166f794209bcef5"} Nov 04 05:02:16.420671 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:16.420634 4395 scope.go:122] "RemoveContainer" containerID="4bb152bc105e6af21a730417de47d0cba974262ca855b8dfe99f7e3ed51bf453" Nov 04 05:02:16.431997 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:16.431980 4395 scope.go:122] "RemoveContainer" containerID="4bb152bc105e6af21a730417de47d0cba974262ca855b8dfe99f7e3ed51bf453" Nov 04 05:02:16.432319 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:02:16.432271 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bb152bc105e6af21a730417de47d0cba974262ca855b8dfe99f7e3ed51bf453\": container with ID starting with 4bb152bc105e6af21a730417de47d0cba974262ca855b8dfe99f7e3ed51bf453 not found: ID does not exist" containerID="4bb152bc105e6af21a730417de47d0cba974262ca855b8dfe99f7e3ed51bf453" Nov 04 05:02:16.432394 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:16.432335 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bb152bc105e6af21a730417de47d0cba974262ca855b8dfe99f7e3ed51bf453"} err="failed to get container status \"4bb152bc105e6af21a730417de47d0cba974262ca855b8dfe99f7e3ed51bf453\": rpc error: code = NotFound desc = could not find container \"4bb152bc105e6af21a730417de47d0cba974262ca855b8dfe99f7e3ed51bf453\": container with ID starting with 4bb152bc105e6af21a730417de47d0cba974262ca855b8dfe99f7e3ed51bf453 not found: ID does not exist" Nov 04 05:02:16.590567 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:16.590537 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c5bb6d/distinct-2-w2-c5bb6d-kserve-78dff78654-r524h"] Nov 04 05:02:16.597436 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:16.597412 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157af145-tt25h" Nov 04 05:02:16.619288 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:16.619255 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-c5bb6d/distinct-2-w2-c5bb6d-kserve-78dff78654-r524h"] Nov 04 05:02:16.624179 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:16.624156 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/906b0195-e1ec-4930-89b1-3b4fb3c1bbdf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/906b0195-e1ec-4930-89b1-3b4fb3c1bbdf-maas-api-xks-ca\") pod \"906b0195-e1ec-4930-89b1-3b4fb3c1bbdf\" (UID: \"906b0195-e1ec-4930-89b1-3b4fb3c1bbdf\") " Nov 04 05:02:16.624284 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:16.624234 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/906b0195-e1ec-4930-89b1-3b4fb3c1bbdf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/906b0195-e1ec-4930-89b1-3b4fb3c1bbdf-maas-api-openshift-ca\") pod \"906b0195-e1ec-4930-89b1-3b4fb3c1bbdf\" (UID: \"906b0195-e1ec-4930-89b1-3b4fb3c1bbdf\") " Nov 04 05:02:16.624401 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:16.624343 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/906b0195-e1ec-4930-89b1-3b4fb3c1bbdf-kube-api-access-xw24c\" (UniqueName: \"kubernetes.io/projected/906b0195-e1ec-4930-89b1-3b4fb3c1bbdf-kube-api-access-xw24c\") pod \"906b0195-e1ec-4930-89b1-3b4fb3c1bbdf\" (UID: \"906b0195-e1ec-4930-89b1-3b4fb3c1bbdf\") " Nov 04 05:02:16.624634 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:16.624608 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/906b0195-e1ec-4930-89b1-3b4fb3c1bbdf-maas-api-openshift-ca" pod "906b0195-e1ec-4930-89b1-3b4fb3c1bbdf" (UID: "906b0195-e1ec-4930-89b1-3b4fb3c1bbdf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:02:16.624728 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:16.624710 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/906b0195-e1ec-4930-89b1-3b4fb3c1bbdf-maas-api-openshift-ca\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:16.626241 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:16.626212 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/906b0195-e1ec-4930-89b1-3b4fb3c1bbdf-maas-api-xks-ca" pod "906b0195-e1ec-4930-89b1-3b4fb3c1bbdf" (UID: "906b0195-e1ec-4930-89b1-3b4fb3c1bbdf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:02:16.626506 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:16.626484 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/906b0195-e1ec-4930-89b1-3b4fb3c1bbdf-kube-api-access-xw24c" pod "906b0195-e1ec-4930-89b1-3b4fb3c1bbdf" (UID: "906b0195-e1ec-4930-89b1-3b4fb3c1bbdf"). InnerVolumeSpecName "kube-api-access-xw24c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:02:16.725985 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:16.725897 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw24c\" (UniqueName: \"kubernetes.io/projected/906b0195-e1ec-4930-89b1-3b4fb3c1bbdf-kube-api-access-xw24c\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:16.725985 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:16.725924 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/906b0195-e1ec-4930-89b1-3b4fb3c1bbdf-maas-api-xks-ca\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:17.068318 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:17.064390 4395 patch_prober.go:29] interesting pod/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 04 05:02:17.068318 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:17.064449 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" podUID="a3d7a106-096e-4551-819f-c8b166554914" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 04 05:02:17.422818 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:17.422740 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157af145-tt25h" Nov 04 05:02:17.422980 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:17.422740 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157af145-tt25h" event={"ID":"906b0195-e1ec-4930-89b1-3b4fb3c1bbdf","Type":"ContainerDied","Data":"a416fc62b98f2ed1e3ceae0c6902b71f94b3e4fb4d135c007d063955e7c83320"} Nov 04 05:02:17.422980 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:17.422876 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a416fc62b98f2ed1e3ceae0c6902b71f94b3e4fb4d135c007d063955e7c83320" Nov 04 05:02:18.067067 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:18.066642 4395 patch_prober.go:29] interesting pod/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 04 05:02:18.067067 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:18.066701 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" podUID="a3d7a106-096e-4551-819f-c8b166554914" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 04 05:02:18.415556 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:18.415482 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f554177-f75c-438e-8d57-71fdd73f7d2e" path="/var/lib/kubelet/pods/8f554177-f75c-438e-8d57-71fdd73f7d2e/volumes" Nov 04 05:02:19.061399 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:19.061364 4395 patch_prober.go:29] interesting pod/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 04 05:02:19.061575 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:19.061416 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" podUID="a3d7a106-096e-4551-819f-c8b166554914" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 04 05:02:19.453788 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:19.453703 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-204e38/distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz"] Nov 04 05:02:19.488375 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:19.488348 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-a48f7e/distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p" Nov 04 05:02:19.507135 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:19.507097 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-a48f7e/distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p" Nov 04 05:02:20.460706 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:20.460675 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a48f7e/distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p" Nov 04 05:02:21.101461 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:21.101424 4395 patch_prober.go:29] interesting pod/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 05:02:21.101666 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:21.101484 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" podUID="a3d7a106-096e-4551-819f-c8b166554914" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 05:02:21.486074 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:21.486004 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" Nov 04 05:02:21.486764 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:21.486736 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" Nov 04 05:02:21.487707 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:21.487687 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" Nov 04 05:02:26.866267 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:26.866235 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n"] Nov 04 05:02:26.866664 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:26.866655 4395 cpu_manager.go:414] "containerMap: removing container" podUID="518e3ddc-9b6e-4127-a47c-397d9117a3ae" containerName="istio-proxy" Nov 04 05:02:26.866709 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:26.866668 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="518e3ddc-9b6e-4127-a47c-397d9117a3ae" containerName="istio-proxy" Nov 04 05:02:26.866709 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:26.866684 4395 cpu_manager.go:414] "containerMap: removing container" podUID="24ea2807-ab22-495c-b66a-b35111e36b97" containerName="istio-proxy" Nov 04 05:02:26.866709 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:26.866689 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24ea2807-ab22-495c-b66a-b35111e36b97" containerName="istio-proxy" Nov 04 05:02:26.866709 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:26.866704 4395 cpu_manager.go:414] "containerMap: removing container" podUID="906b0195-e1ec-4930-89b1-3b4fb3c1bbdf" containerName="revoke-keys" Nov 04 05:02:26.866709 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:26.866709 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="906b0195-e1ec-4930-89b1-3b4fb3c1bbdf" containerName="revoke-keys" Nov 04 05:02:26.866853 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:26.866723 4395 cpu_manager.go:414] "containerMap: removing container" podUID="8f554177-f75c-438e-8d57-71fdd73f7d2e" containerName="main" Nov 04 05:02:26.866853 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:26.866728 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f554177-f75c-438e-8d57-71fdd73f7d2e" containerName="main" Nov 04 05:02:26.866853 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:26.866810 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="906b0195-e1ec-4930-89b1-3b4fb3c1bbdf" containerName="revoke-keys" Nov 04 05:02:26.866853 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:26.866820 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="518e3ddc-9b6e-4127-a47c-397d9117a3ae" containerName="istio-proxy" Nov 04 05:02:26.866853 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:26.866827 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f554177-f75c-438e-8d57-71fdd73f7d2e" containerName="main" Nov 04 05:02:26.866853 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:26.866834 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="24ea2807-ab22-495c-b66a-b35111e36b97" containerName="istio-proxy" Nov 04 05:02:26.866853 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:26.866842 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="24ea2807-ab22-495c-b66a-b35111e36b97" containerName="istio-proxy" Nov 04 05:02:26.871238 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:26.871212 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" Nov 04 05:02:26.949794 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:26.949754 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fed21ce4-c40a-4b31-87cf-bf331f4649b8-workload-certs\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n\" (UID: \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" Nov 04 05:02:26.950111 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:26.950083 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fed21ce4-c40a-4b31-87cf-bf331f4649b8-istio-token\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n\" (UID: \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" Nov 04 05:02:26.950248 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:26.950137 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fed21ce4-c40a-4b31-87cf-bf331f4649b8-istiod-ca-cert\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n\" (UID: \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" Nov 04 05:02:26.950248 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:26.950174 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed21ce4-c40a-4b31-87cf-bf331f4649b8-credential-socket\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n\" (UID: \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" Nov 04 05:02:26.950248 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:26.950199 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fed21ce4-c40a-4b31-87cf-bf331f4649b8-istio-data\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n\" (UID: \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" Nov 04 05:02:26.950454 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:26.950262 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed21ce4-c40a-4b31-87cf-bf331f4649b8-workload-socket\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n\" (UID: \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" Nov 04 05:02:26.950454 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:26.950351 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fed21ce4-c40a-4b31-87cf-bf331f4649b8-istio-envoy\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n\" (UID: \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" Nov 04 05:02:26.950454 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:26.950403 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fed21ce4-c40a-4b31-87cf-bf331f4649b8-istio-podinfo\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n\" (UID: \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" Nov 04 05:02:26.950454 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:26.950443 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7wkk\" (UniqueName: \"kubernetes.io/projected/fed21ce4-c40a-4b31-87cf-bf331f4649b8-kube-api-access-d7wkk\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n\" (UID: \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" Nov 04 05:02:26.971780 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:26.971753 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n"] Nov 04 05:02:27.051257 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.051208 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7wkk\" (UniqueName: \"kubernetes.io/projected/fed21ce4-c40a-4b31-87cf-bf331f4649b8-kube-api-access-d7wkk\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n\" (UID: \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" Nov 04 05:02:27.051462 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.051334 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fed21ce4-c40a-4b31-87cf-bf331f4649b8-workload-certs\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n\" (UID: \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" Nov 04 05:02:27.051462 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.051399 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fed21ce4-c40a-4b31-87cf-bf331f4649b8-istio-token\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n\" (UID: \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" Nov 04 05:02:27.051462 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.051433 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fed21ce4-c40a-4b31-87cf-bf331f4649b8-istiod-ca-cert\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n\" (UID: \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" Nov 04 05:02:27.051653 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.051464 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed21ce4-c40a-4b31-87cf-bf331f4649b8-credential-socket\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n\" (UID: \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" Nov 04 05:02:27.051653 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.051491 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fed21ce4-c40a-4b31-87cf-bf331f4649b8-istio-data\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n\" (UID: \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" Nov 04 05:02:27.051653 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.051560 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed21ce4-c40a-4b31-87cf-bf331f4649b8-workload-socket\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n\" (UID: \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" Nov 04 05:02:27.051653 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.051615 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fed21ce4-c40a-4b31-87cf-bf331f4649b8-istio-envoy\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n\" (UID: \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" Nov 04 05:02:27.051817 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.051666 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fed21ce4-c40a-4b31-87cf-bf331f4649b8-istio-podinfo\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n\" (UID: \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" Nov 04 05:02:27.051817 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.051701 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fed21ce4-c40a-4b31-87cf-bf331f4649b8-workload-certs\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n\" (UID: \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" Nov 04 05:02:27.052066 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.052040 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fed21ce4-c40a-4b31-87cf-bf331f4649b8-istio-data\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n\" (UID: \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" Nov 04 05:02:27.052176 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.052156 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed21ce4-c40a-4b31-87cf-bf331f4649b8-workload-socket\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n\" (UID: \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" Nov 04 05:02:27.052227 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.052178 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed21ce4-c40a-4b31-87cf-bf331f4649b8-credential-socket\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n\" (UID: \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" Nov 04 05:02:27.052352 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.052330 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fed21ce4-c40a-4b31-87cf-bf331f4649b8-istiod-ca-cert\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n\" (UID: \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" Nov 04 05:02:27.054168 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.054138 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fed21ce4-c40a-4b31-87cf-bf331f4649b8-istio-envoy\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n\" (UID: \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" Nov 04 05:02:27.054293 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.054260 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fed21ce4-c40a-4b31-87cf-bf331f4649b8-istio-podinfo\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n\" (UID: \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" Nov 04 05:02:27.108292 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.108263 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fed21ce4-c40a-4b31-87cf-bf331f4649b8-istio-token\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n\" (UID: \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" Nov 04 05:02:27.108628 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.108611 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7wkk\" (UniqueName: \"kubernetes.io/projected/fed21ce4-c40a-4b31-87cf-bf331f4649b8-kube-api-access-d7wkk\") pod \"e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n\" (UID: \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\") " pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" Nov 04 05:02:27.185938 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.185864 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" Nov 04 05:02:27.257668 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.256385 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g"] Nov 04 05:02:27.257668 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.257025 4395 cpu_manager.go:414] "containerMap: removing container" podUID="24ea2807-ab22-495c-b66a-b35111e36b97" containerName="istio-proxy" Nov 04 05:02:27.257668 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.257042 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24ea2807-ab22-495c-b66a-b35111e36b97" containerName="istio-proxy" Nov 04 05:02:27.263077 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.262720 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" Nov 04 05:02:27.321048 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.321010 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g"] Nov 04 05:02:27.355551 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.355522 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54c856ac-40f6-4c13-8ae5-8282967b7496-istio-token\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g\" (UID: \"54c856ac-40f6-4c13-8ae5-8282967b7496\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" Nov 04 05:02:27.355704 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.355555 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c856ac-40f6-4c13-8ae5-8282967b7496-istio-envoy\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g\" (UID: \"54c856ac-40f6-4c13-8ae5-8282967b7496\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" Nov 04 05:02:27.355704 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.355589 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c856ac-40f6-4c13-8ae5-8282967b7496-istio-podinfo\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g\" (UID: \"54c856ac-40f6-4c13-8ae5-8282967b7496\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" Nov 04 05:02:27.355838 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.355759 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c856ac-40f6-4c13-8ae5-8282967b7496-workload-socket\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g\" (UID: \"54c856ac-40f6-4c13-8ae5-8282967b7496\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" Nov 04 05:02:27.355838 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.355808 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl7gj\" (UniqueName: \"kubernetes.io/projected/54c856ac-40f6-4c13-8ae5-8282967b7496-kube-api-access-pl7gj\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g\" (UID: \"54c856ac-40f6-4c13-8ae5-8282967b7496\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" Nov 04 05:02:27.355942 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.355922 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c856ac-40f6-4c13-8ae5-8282967b7496-workload-certs\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g\" (UID: \"54c856ac-40f6-4c13-8ae5-8282967b7496\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" Nov 04 05:02:27.356001 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.355951 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c856ac-40f6-4c13-8ae5-8282967b7496-istiod-ca-cert\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g\" (UID: \"54c856ac-40f6-4c13-8ae5-8282967b7496\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" Nov 04 05:02:27.356001 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.355979 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c856ac-40f6-4c13-8ae5-8282967b7496-istio-data\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g\" (UID: \"54c856ac-40f6-4c13-8ae5-8282967b7496\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" Nov 04 05:02:27.356110 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.356070 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c856ac-40f6-4c13-8ae5-8282967b7496-credential-socket\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g\" (UID: \"54c856ac-40f6-4c13-8ae5-8282967b7496\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" Nov 04 05:02:27.457511 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.457428 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c856ac-40f6-4c13-8ae5-8282967b7496-istio-podinfo\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g\" (UID: \"54c856ac-40f6-4c13-8ae5-8282967b7496\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" Nov 04 05:02:27.457511 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.457488 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c856ac-40f6-4c13-8ae5-8282967b7496-workload-socket\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g\" (UID: \"54c856ac-40f6-4c13-8ae5-8282967b7496\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" Nov 04 05:02:27.457511 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.457506 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl7gj\" (UniqueName: \"kubernetes.io/projected/54c856ac-40f6-4c13-8ae5-8282967b7496-kube-api-access-pl7gj\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g\" (UID: \"54c856ac-40f6-4c13-8ae5-8282967b7496\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" Nov 04 05:02:27.457781 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.457555 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c856ac-40f6-4c13-8ae5-8282967b7496-workload-certs\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g\" (UID: \"54c856ac-40f6-4c13-8ae5-8282967b7496\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" Nov 04 05:02:27.457781 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.457584 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c856ac-40f6-4c13-8ae5-8282967b7496-istiod-ca-cert\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g\" (UID: \"54c856ac-40f6-4c13-8ae5-8282967b7496\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" Nov 04 05:02:27.457781 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.457609 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c856ac-40f6-4c13-8ae5-8282967b7496-istio-data\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g\" (UID: \"54c856ac-40f6-4c13-8ae5-8282967b7496\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" Nov 04 05:02:27.457781 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.457642 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c856ac-40f6-4c13-8ae5-8282967b7496-credential-socket\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g\" (UID: \"54c856ac-40f6-4c13-8ae5-8282967b7496\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" Nov 04 05:02:27.457781 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.457724 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54c856ac-40f6-4c13-8ae5-8282967b7496-istio-token\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g\" (UID: \"54c856ac-40f6-4c13-8ae5-8282967b7496\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" Nov 04 05:02:27.457781 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.457753 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c856ac-40f6-4c13-8ae5-8282967b7496-istio-envoy\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g\" (UID: \"54c856ac-40f6-4c13-8ae5-8282967b7496\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" Nov 04 05:02:27.458092 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.457924 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c856ac-40f6-4c13-8ae5-8282967b7496-workload-socket\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g\" (UID: \"54c856ac-40f6-4c13-8ae5-8282967b7496\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" Nov 04 05:02:27.458092 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.458047 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c856ac-40f6-4c13-8ae5-8282967b7496-workload-certs\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g\" (UID: \"54c856ac-40f6-4c13-8ae5-8282967b7496\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" Nov 04 05:02:27.458335 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.458292 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c856ac-40f6-4c13-8ae5-8282967b7496-istio-data\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g\" (UID: \"54c856ac-40f6-4c13-8ae5-8282967b7496\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" Nov 04 05:02:27.458472 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.458353 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c856ac-40f6-4c13-8ae5-8282967b7496-credential-socket\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g\" (UID: \"54c856ac-40f6-4c13-8ae5-8282967b7496\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" Nov 04 05:02:27.458613 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.458527 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c856ac-40f6-4c13-8ae5-8282967b7496-istiod-ca-cert\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g\" (UID: \"54c856ac-40f6-4c13-8ae5-8282967b7496\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" Nov 04 05:02:27.460287 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.460263 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c856ac-40f6-4c13-8ae5-8282967b7496-istio-envoy\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g\" (UID: \"54c856ac-40f6-4c13-8ae5-8282967b7496\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" Nov 04 05:02:27.460413 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.460337 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c856ac-40f6-4c13-8ae5-8282967b7496-istio-podinfo\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g\" (UID: \"54c856ac-40f6-4c13-8ae5-8282967b7496\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" Nov 04 05:02:27.514443 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.514413 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54c856ac-40f6-4c13-8ae5-8282967b7496-istio-token\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g\" (UID: \"54c856ac-40f6-4c13-8ae5-8282967b7496\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" Nov 04 05:02:27.514822 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.514799 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl7gj\" (UniqueName: \"kubernetes.io/projected/54c856ac-40f6-4c13-8ae5-8282967b7496-kube-api-access-pl7gj\") pod \"e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g\" (UID: \"54c856ac-40f6-4c13-8ae5-8282967b7496\") " pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" Nov 04 05:02:27.584896 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.584869 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" Nov 04 05:02:27.624673 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.624581 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n"] Nov 04 05:02:27.629910 ip-10-0-134-224 kubenswrapper[4395]: W1104 05:02:27.629878 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfed21ce4_c40a_4b31_87cf_bf331f4649b8.slice/crio-028230f753e72da514efd123cf46a9ccf361254871b12f9a7f942c54394268fe WatchSource:0}: Error finding container 028230f753e72da514efd123cf46a9ccf361254871b12f9a7f942c54394268fe: Status 404 returned error can't find the container with id 028230f753e72da514efd123cf46a9ccf361254871b12f9a7f942c54394268fe Nov 04 05:02:27.634050 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.634011 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 05:02:27.634122 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.634099 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 05:02:27.634167 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.634140 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 05:02:27.969961 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.969939 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g"] Nov 04 05:02:27.973007 ip-10-0-134-224 kubenswrapper[4395]: W1104 05:02:27.972981 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54c856ac_40f6_4c13_8ae5_8282967b7496.slice/crio-fabe6da8a84f88dd3b2e737d5145ce493ce41164d42933f0a6582e285e7ed839 WatchSource:0}: Error finding container fabe6da8a84f88dd3b2e737d5145ce493ce41164d42933f0a6582e285e7ed839: Status 404 returned error can't find the container with id fabe6da8a84f88dd3b2e737d5145ce493ce41164d42933f0a6582e285e7ed839 Nov 04 05:02:27.976174 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.976110 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 05:02:27.976276 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.976254 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 05:02:27.976338 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:27.976325 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 05:02:28.532534 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:28.532488 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" event={"ID":"fed21ce4-c40a-4b31-87cf-bf331f4649b8","Type":"ContainerStarted","Data":"b4a06fc077e7beb6572b9601846a5ab3e98fb6d4c850b028842a738fa79f4b56"} Nov 04 05:02:28.532738 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:28.532592 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" event={"ID":"fed21ce4-c40a-4b31-87cf-bf331f4649b8","Type":"ContainerStarted","Data":"028230f753e72da514efd123cf46a9ccf361254871b12f9a7f942c54394268fe"} Nov 04 05:02:28.535257 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:28.535224 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" event={"ID":"54c856ac-40f6-4c13-8ae5-8282967b7496","Type":"ContainerStarted","Data":"0fa8f8564f87f678aee07f25dad5c0d576a45af16729581f2c02447c491eeff9"} Nov 04 05:02:28.535440 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:28.535283 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" event={"ID":"54c856ac-40f6-4c13-8ae5-8282967b7496","Type":"ContainerStarted","Data":"fabe6da8a84f88dd3b2e737d5145ce493ce41164d42933f0a6582e285e7ed839"} Nov 04 05:02:28.590322 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:28.587474 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" Nov 04 05:02:28.809383 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:28.808152 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" podStartSLOduration=1.808129705 podStartE2EDuration="1.808129705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:02:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:02:28.786145655 +0000 UTC m=+990.476228103" watchObservedRunningTime="2026-11-04 05:02:28.808129705 +0000 UTC m=+990.498212151" Nov 04 05:02:28.809810 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:28.809764 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" podStartSLOduration=2.8097520769999997 podStartE2EDuration="2.809752077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:02:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:02:28.673698673 +0000 UTC m=+990.363781159" watchObservedRunningTime="2026-11-04 05:02:28.809752077 +0000 UTC m=+990.499834526" Nov 04 05:02:29.192372 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:29.191450 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" Nov 04 05:02:29.205918 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:29.204427 4395 patch_prober.go:29] interesting pod/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 04 05:02:29.205918 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:29.204496 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" podUID="fed21ce4-c40a-4b31-87cf-bf331f4649b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 04 05:02:29.605323 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:29.602436 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 04 05:02:29.605323 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:29.602512 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" podUID="54c856ac-40f6-4c13-8ae5-8282967b7496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 04 05:02:30.187321 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:30.186650 4395 patch_prober.go:29] interesting pod/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 04 05:02:30.187321 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:30.186709 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" podUID="fed21ce4-c40a-4b31-87cf-bf331f4649b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 04 05:02:30.591323 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:30.587373 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 04 05:02:30.591323 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:30.587428 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" podUID="54c856ac-40f6-4c13-8ae5-8282967b7496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 04 05:02:31.148820 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:31.148782 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-204e38-7479dcc588-qhznn"] Nov 04 05:02:31.192735 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:31.191887 4395 patch_prober.go:29] interesting pod/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 04 05:02:31.192735 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:31.191938 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" podUID="fed21ce4-c40a-4b31-87cf-bf331f4649b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 04 05:02:31.550693 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:31.550660 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-919dcb68-cpk9z"] Nov 04 05:02:31.555227 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:31.555204 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-919dcb68-cpk9z" Nov 04 05:02:31.608320 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:31.588724 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 04 05:02:31.608320 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:31.588772 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" podUID="54c856ac-40f6-4c13-8ae5-8282967b7496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 04 05:02:31.622517 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:31.612200 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 05:02:31.622517 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:31.619688 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nz4vh\"" Nov 04 05:02:31.622517 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:31.619747 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 05:02:31.622517 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:31.619810 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 05:02:31.629667 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:31.624460 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-919dcb68-cpk9z"] Nov 04 05:02:31.643485 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:31.641395 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/029b0ad5-7427-432a-8f07-03c6646a300a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-919dcb68-cpk9z\" (UID: \"029b0ad5-7427-432a-8f07-03c6646a300a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-919dcb68-cpk9z" Nov 04 05:02:31.643485 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:31.641533 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2zmv\" (UniqueName: \"kubernetes.io/projected/029b0ad5-7427-432a-8f07-03c6646a300a-kube-api-access-w2zmv\") pod \"maas-api-revoke-sub-919dcb68-cpk9z\" (UID: \"029b0ad5-7427-432a-8f07-03c6646a300a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-919dcb68-cpk9z" Nov 04 05:02:31.643485 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:31.641642 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/029b0ad5-7427-432a-8f07-03c6646a300a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-919dcb68-cpk9z\" (UID: \"029b0ad5-7427-432a-8f07-03c6646a300a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-919dcb68-cpk9z" Nov 04 05:02:31.746070 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:31.744162 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/029b0ad5-7427-432a-8f07-03c6646a300a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-919dcb68-cpk9z\" (UID: \"029b0ad5-7427-432a-8f07-03c6646a300a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-919dcb68-cpk9z" Nov 04 05:02:31.746070 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:31.744275 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2zmv\" (UniqueName: \"kubernetes.io/projected/029b0ad5-7427-432a-8f07-03c6646a300a-kube-api-access-w2zmv\") pod \"maas-api-revoke-sub-919dcb68-cpk9z\" (UID: \"029b0ad5-7427-432a-8f07-03c6646a300a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-919dcb68-cpk9z" Nov 04 05:02:31.746070 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:31.744416 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/029b0ad5-7427-432a-8f07-03c6646a300a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-919dcb68-cpk9z\" (UID: \"029b0ad5-7427-432a-8f07-03c6646a300a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-919dcb68-cpk9z" Nov 04 05:02:31.746070 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:31.744996 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/029b0ad5-7427-432a-8f07-03c6646a300a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-919dcb68-cpk9z\" (UID: \"029b0ad5-7427-432a-8f07-03c6646a300a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-919dcb68-cpk9z" Nov 04 05:02:31.747708 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:31.747670 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/029b0ad5-7427-432a-8f07-03c6646a300a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-919dcb68-cpk9z\" (UID: \"029b0ad5-7427-432a-8f07-03c6646a300a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-919dcb68-cpk9z" Nov 04 05:02:31.811890 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:31.811816 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2zmv\" (UniqueName: \"kubernetes.io/projected/029b0ad5-7427-432a-8f07-03c6646a300a-kube-api-access-w2zmv\") pod \"maas-api-revoke-sub-919dcb68-cpk9z\" (UID: \"029b0ad5-7427-432a-8f07-03c6646a300a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-919dcb68-cpk9z" Nov 04 05:02:31.921390 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:31.921235 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-919dcb68-cpk9z" Nov 04 05:02:32.192374 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:32.191386 4395 patch_prober.go:29] interesting pod/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 04 05:02:32.192374 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:32.191463 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" podUID="fed21ce4-c40a-4b31-87cf-bf331f4649b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 04 05:02:32.412584 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:32.409342 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-919dcb68-cpk9z"] Nov 04 05:02:32.573350 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:32.573186 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-919dcb68-cpk9z" event={"ID":"029b0ad5-7427-432a-8f07-03c6646a300a","Type":"ContainerStarted","Data":"7c82010c26e8fba9435f728ee218a3a9c61f3bfe66e064c9a2384c565810711c"} Nov 04 05:02:32.595028 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:32.594624 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 04 05:02:32.595028 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:32.594677 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" podUID="54c856ac-40f6-4c13-8ae5-8282967b7496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 04 05:02:33.190119 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:33.189695 4395 patch_prober.go:29] interesting pod/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 04 05:02:33.190119 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:33.189755 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" podUID="fed21ce4-c40a-4b31-87cf-bf331f4649b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 04 05:02:33.485372 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:02:33.482558 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 05:02:33.546081 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:02:33.546048 4395 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod029b0ad5_7427_432a_8f07_03c6646a300a.slice/crio-3600c62e74dd92f1b5a6c5aef3e07e9d20e5210e534257d96089aa3838e4a6b7.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod029b0ad5_7427_432a_8f07_03c6646a300a.slice/crio-conmon-3600c62e74dd92f1b5a6c5aef3e07e9d20e5210e534257d96089aa3838e4a6b7.scope\": RecentStats: unable to find data in memory cache]" Nov 04 05:02:33.593690 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:33.593651 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 04 05:02:33.593939 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:33.593908 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" podUID="54c856ac-40f6-4c13-8ae5-8282967b7496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 04 05:02:33.609932 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:33.609900 4395 generic.go:356] "Generic (PLEG): container finished" podID="029b0ad5-7427-432a-8f07-03c6646a300a" containerID="3600c62e74dd92f1b5a6c5aef3e07e9d20e5210e534257d96089aa3838e4a6b7" exitCode=0 Nov 04 05:02:33.610260 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:33.610243 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-919dcb68-cpk9z" event={"ID":"029b0ad5-7427-432a-8f07-03c6646a300a","Type":"ContainerDied","Data":"3600c62e74dd92f1b5a6c5aef3e07e9d20e5210e534257d96089aa3838e4a6b7"} Nov 04 05:02:34.187347 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:34.187294 4395 patch_prober.go:29] interesting pod/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 04 05:02:34.187500 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:34.187373 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" podUID="fed21ce4-c40a-4b31-87cf-bf331f4649b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 04 05:02:34.476257 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:34.476168 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-204e38/distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz" podUID="6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1" containerName="main" containerID="cri-o://b4107181df7b48600429c85710c61f571d0912a131c5232e70ab36117d2b1900" gracePeriod=45 Nov 04 05:02:34.585508 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:34.585468 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 04 05:02:34.585663 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:34.585544 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" podUID="54c856ac-40f6-4c13-8ae5-8282967b7496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 04 05:02:34.618906 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:34.618870 4395 generic.go:356] "Generic (PLEG): container finished" podID="6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1" containerID="b4107181df7b48600429c85710c61f571d0912a131c5232e70ab36117d2b1900" exitCode=0 Nov 04 05:02:34.619029 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:34.618940 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-204e38/distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz" event={"ID":"6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1","Type":"ContainerDied","Data":"b4107181df7b48600429c85710c61f571d0912a131c5232e70ab36117d2b1900"} Nov 04 05:02:34.768375 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:34.767496 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-204e38/distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz" Nov 04 05:02:34.844516 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:34.844483 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-919dcb68-cpk9z" Nov 04 05:02:34.909870 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:34.909838 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-model-cache\") pod \"6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1\" (UID: \"6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1\") " Nov 04 05:02:34.910017 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:34.909919 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-kube-api-access-fpqsl\" (UniqueName: \"kubernetes.io/projected/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-kube-api-access-fpqsl\") pod \"6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1\" (UID: \"6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1\") " Nov 04 05:02:34.910017 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:34.909973 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-tmp-dir\") pod \"6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1\" (UID: \"6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1\") " Nov 04 05:02:34.910153 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:34.910019 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-tls-certs\") pod \"6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1\" (UID: \"6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1\") " Nov 04 05:02:34.910153 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:34.910080 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-home\" (UniqueName: \"kubernetes.io/empty-dir/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-home\") pod \"6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1\" (UID: \"6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1\") " Nov 04 05:02:34.910153 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:34.910112 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-dshm\") pod \"6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1\" (UID: \"6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1\") " Nov 04 05:02:34.910354 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:34.910281 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-tmp-dir" pod "6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1" (UID: "6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:34.910526 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:34.910504 4395 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-tmp-dir\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:34.910634 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:34.910594 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-home" pod "6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1" (UID: "6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:34.910700 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:34.910687 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-model-cache" pod "6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1" (UID: "6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:34.913031 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:34.913004 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-dshm" pod "6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1" (UID: "6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:34.913144 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:34.913038 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-tls-certs" pod "6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1" (UID: "6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:02:34.913429 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:34.913401 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-kube-api-access-fpqsl" pod "6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1" (UID: "6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1"). InnerVolumeSpecName "kube-api-access-fpqsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:02:35.012808 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:35.011676 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/029b0ad5-7427-432a-8f07-03c6646a300a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/029b0ad5-7427-432a-8f07-03c6646a300a-maas-api-openshift-ca\") pod \"029b0ad5-7427-432a-8f07-03c6646a300a\" (UID: \"029b0ad5-7427-432a-8f07-03c6646a300a\") " Nov 04 05:02:35.012808 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:35.011786 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/029b0ad5-7427-432a-8f07-03c6646a300a-kube-api-access-w2zmv\" (UniqueName: \"kubernetes.io/projected/029b0ad5-7427-432a-8f07-03c6646a300a-kube-api-access-w2zmv\") pod \"029b0ad5-7427-432a-8f07-03c6646a300a\" (UID: \"029b0ad5-7427-432a-8f07-03c6646a300a\") " Nov 04 05:02:35.012808 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:35.011876 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/029b0ad5-7427-432a-8f07-03c6646a300a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/029b0ad5-7427-432a-8f07-03c6646a300a-maas-api-xks-ca\") pod \"029b0ad5-7427-432a-8f07-03c6646a300a\" (UID: \"029b0ad5-7427-432a-8f07-03c6646a300a\") " Nov 04 05:02:35.012808 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:35.012275 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpqsl\" (UniqueName: \"kubernetes.io/projected/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-kube-api-access-fpqsl\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:35.012808 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:35.012369 4395 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-tls-certs\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:35.012808 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:35.012390 4395 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-home\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:35.012808 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:35.012405 4395 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-dshm\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:35.012808 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:35.012417 4395 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1-model-cache\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:35.013419 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:35.013052 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/029b0ad5-7427-432a-8f07-03c6646a300a-maas-api-openshift-ca" pod "029b0ad5-7427-432a-8f07-03c6646a300a" (UID: "029b0ad5-7427-432a-8f07-03c6646a300a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:02:35.015714 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:35.015677 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/029b0ad5-7427-432a-8f07-03c6646a300a-maas-api-xks-ca" pod "029b0ad5-7427-432a-8f07-03c6646a300a" (UID: "029b0ad5-7427-432a-8f07-03c6646a300a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:02:35.016458 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:35.016424 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/029b0ad5-7427-432a-8f07-03c6646a300a-kube-api-access-w2zmv" pod "029b0ad5-7427-432a-8f07-03c6646a300a" (UID: "029b0ad5-7427-432a-8f07-03c6646a300a"). InnerVolumeSpecName "kube-api-access-w2zmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:02:35.114373 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:35.114334 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/029b0ad5-7427-432a-8f07-03c6646a300a-maas-api-xks-ca\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:35.114373 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:35.114379 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/029b0ad5-7427-432a-8f07-03c6646a300a-maas-api-openshift-ca\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:35.114612 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:35.114396 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2zmv\" (UniqueName: \"kubernetes.io/projected/029b0ad5-7427-432a-8f07-03c6646a300a-kube-api-access-w2zmv\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:35.626550 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:35.626508 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-204e38/distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz" Nov 04 05:02:35.626550 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:35.626530 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-204e38/distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz" event={"ID":"6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1","Type":"ContainerDied","Data":"a21b42ba6148bb027b165803d79611a29520621311f72a05e3425e5da90dcd5a"} Nov 04 05:02:35.627071 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:35.626587 4395 scope.go:122] "RemoveContainer" containerID="b4107181df7b48600429c85710c61f571d0912a131c5232e70ab36117d2b1900" Nov 04 05:02:35.628042 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:35.628014 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-919dcb68-cpk9z" event={"ID":"029b0ad5-7427-432a-8f07-03c6646a300a","Type":"ContainerDied","Data":"7c82010c26e8fba9435f728ee218a3a9c61f3bfe66e064c9a2384c565810711c"} Nov 04 05:02:35.628126 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:35.628066 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c82010c26e8fba9435f728ee218a3a9c61f3bfe66e064c9a2384c565810711c" Nov 04 05:02:35.628181 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:35.628131 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-919dcb68-cpk9z" Nov 04 05:02:35.708513 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:35.708477 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" Nov 04 05:02:35.709023 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:35.708998 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" Nov 04 05:02:35.709977 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:35.709958 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" Nov 04 05:02:35.829396 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:35.829363 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-204e38/distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz"] Nov 04 05:02:35.875112 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:35.875079 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-204e38/distinct-2-w3-204e38-kserve-655bbd54cf-vfrkz"] Nov 04 05:02:36.149581 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:36.149477 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-204e38-7479dcc588-qhznn" podUID="1b78fb66-665e-454a-bc99-7331cfaf9bdc" containerName="payload-processing" containerID="cri-o://ec74d9dc4f9dc53e081e51ed53ea9fa1c81f164e14eef4711e10e643741bfd38" gracePeriod=25 Nov 04 05:02:36.187163 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:36.187121 4395 patch_prober.go:29] interesting pod/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 05:02:36.187330 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:36.187178 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" podUID="fed21ce4-c40a-4b31-87cf-bf331f4649b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 05:02:36.443617 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:36.442364 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1" path="/var/lib/kubelet/pods/6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1/volumes" Nov 04 05:02:36.444777 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:36.444756 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-204e38-7479dcc588-qhznn" Nov 04 05:02:36.529354 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:36.529293 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b78fb66-665e-454a-bc99-7331cfaf9bdc-kube-api-access-sc46b\" (UniqueName: \"kubernetes.io/projected/1b78fb66-665e-454a-bc99-7331cfaf9bdc-kube-api-access-sc46b\") pod \"1b78fb66-665e-454a-bc99-7331cfaf9bdc\" (UID: \"1b78fb66-665e-454a-bc99-7331cfaf9bdc\") " Nov 04 05:02:36.529475 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:36.529449 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b78fb66-665e-454a-bc99-7331cfaf9bdc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b78fb66-665e-454a-bc99-7331cfaf9bdc-plugins-config-volume\") pod \"1b78fb66-665e-454a-bc99-7331cfaf9bdc\" (UID: \"1b78fb66-665e-454a-bc99-7331cfaf9bdc\") " Nov 04 05:02:36.529968 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:36.529942 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b78fb66-665e-454a-bc99-7331cfaf9bdc-plugins-config-volume" pod "1b78fb66-665e-454a-bc99-7331cfaf9bdc" (UID: "1b78fb66-665e-454a-bc99-7331cfaf9bdc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:02:36.531436 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:36.531412 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b78fb66-665e-454a-bc99-7331cfaf9bdc-kube-api-access-sc46b" pod "1b78fb66-665e-454a-bc99-7331cfaf9bdc" (UID: "1b78fb66-665e-454a-bc99-7331cfaf9bdc"). InnerVolumeSpecName "kube-api-access-sc46b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:02:36.631112 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:36.631079 4395 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b78fb66-665e-454a-bc99-7331cfaf9bdc-plugins-config-volume\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:36.631112 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:36.631109 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc46b\" (UniqueName: \"kubernetes.io/projected/1b78fb66-665e-454a-bc99-7331cfaf9bdc-kube-api-access-sc46b\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:36.637743 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:36.637713 4395 generic.go:356] "Generic (PLEG): container finished" podID="1b78fb66-665e-454a-bc99-7331cfaf9bdc" containerID="ec74d9dc4f9dc53e081e51ed53ea9fa1c81f164e14eef4711e10e643741bfd38" exitCode=0 Nov 04 05:02:36.637881 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:36.637782 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-204e38-7479dcc588-qhznn" Nov 04 05:02:36.637881 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:36.637835 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-204e38-7479dcc588-qhznn" event={"ID":"1b78fb66-665e-454a-bc99-7331cfaf9bdc","Type":"ContainerDied","Data":"ec74d9dc4f9dc53e081e51ed53ea9fa1c81f164e14eef4711e10e643741bfd38"} Nov 04 05:02:36.637881 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:36.637877 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-204e38-7479dcc588-qhznn" event={"ID":"1b78fb66-665e-454a-bc99-7331cfaf9bdc","Type":"ContainerDied","Data":"626f46310121904c4be0d621972c3f3ba49f169dfd1d5321e343800681577c6d"} Nov 04 05:02:36.638065 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:36.637897 4395 scope.go:122] "RemoveContainer" containerID="ec74d9dc4f9dc53e081e51ed53ea9fa1c81f164e14eef4711e10e643741bfd38" Nov 04 05:02:36.648398 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:36.648378 4395 scope.go:122] "RemoveContainer" containerID="ec74d9dc4f9dc53e081e51ed53ea9fa1c81f164e14eef4711e10e643741bfd38" Nov 04 05:02:36.648732 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:02:36.648704 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec74d9dc4f9dc53e081e51ed53ea9fa1c81f164e14eef4711e10e643741bfd38\": container with ID starting with ec74d9dc4f9dc53e081e51ed53ea9fa1c81f164e14eef4711e10e643741bfd38 not found: ID does not exist" containerID="ec74d9dc4f9dc53e081e51ed53ea9fa1c81f164e14eef4711e10e643741bfd38" Nov 04 05:02:36.648811 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:36.648741 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec74d9dc4f9dc53e081e51ed53ea9fa1c81f164e14eef4711e10e643741bfd38"} err="failed to get container status \"ec74d9dc4f9dc53e081e51ed53ea9fa1c81f164e14eef4711e10e643741bfd38\": rpc error: code = NotFound desc = could not find container \"ec74d9dc4f9dc53e081e51ed53ea9fa1c81f164e14eef4711e10e643741bfd38\": container with ID starting with ec74d9dc4f9dc53e081e51ed53ea9fa1c81f164e14eef4711e10e643741bfd38 not found: ID does not exist" Nov 04 05:02:36.813674 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:36.811389 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-204e38-7479dcc588-qhznn"] Nov 04 05:02:36.813674 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:36.813570 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-204e38-7479dcc588-qhznn"] Nov 04 05:02:37.082481 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:37.082391 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" Nov 04 05:02:37.082861 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:37.082834 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" Nov 04 05:02:37.085177 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:37.085153 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" Nov 04 05:02:38.416437 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:38.416408 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b78fb66-665e-454a-bc99-7331cfaf9bdc" path="/var/lib/kubelet/pods/1b78fb66-665e-454a-bc99-7331cfaf9bdc/volumes" Nov 04 05:02:45.152178 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:45.152140 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n"] Nov 04 05:02:45.152935 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:45.152370 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" podUID="fed21ce4-c40a-4b31-87cf-bf331f4649b8" containerName="istio-proxy" containerID="cri-o://b4a06fc077e7beb6572b9601846a5ab3e98fb6d4c850b028842a738fa79f4b56" gracePeriod=30 Nov 04 05:02:45.173737 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:45.173703 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5"] Nov 04 05:02:45.173953 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:45.173926 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" podUID="a3d7a106-096e-4551-819f-c8b166554914" containerName="istio-proxy" containerID="cri-o://2bbbefde8fe524095f1c456d56cdb7dbc0abb470b208f02d14cea8700557cfac" gracePeriod=30 Nov 04 05:02:45.191913 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:45.191881 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r"] Nov 04 05:02:45.192134 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:45.192091 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" podUID="11dc70f6-5166-4ec0-a746-9d75ee2c6e56" containerName="istio-proxy" containerID="cri-o://1fc2684f299796b9b55ae5a33809495dfe29551cd03fcd9f67195045d9e76417" gracePeriod=30 Nov 04 05:02:45.849602 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:45.849566 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad6d2d59-8hwjz"] Nov 04 05:02:45.850254 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:45.850228 4395 cpu_manager.go:414] "containerMap: removing container" podUID="029b0ad5-7427-432a-8f07-03c6646a300a" containerName="revoke-keys" Nov 04 05:02:45.850254 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:45.850255 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="029b0ad5-7427-432a-8f07-03c6646a300a" containerName="revoke-keys" Nov 04 05:02:45.850442 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:45.850288 4395 cpu_manager.go:414] "containerMap: removing container" podUID="1b78fb66-665e-454a-bc99-7331cfaf9bdc" containerName="payload-processing" Nov 04 05:02:45.850442 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:45.850313 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b78fb66-665e-454a-bc99-7331cfaf9bdc" containerName="payload-processing" Nov 04 05:02:45.850442 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:45.850361 4395 cpu_manager.go:414] "containerMap: removing container" podUID="6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1" containerName="main" Nov 04 05:02:45.850442 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:45.850369 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1" containerName="main" Nov 04 05:02:45.850569 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:45.850466 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b78fb66-665e-454a-bc99-7331cfaf9bdc" containerName="payload-processing" Nov 04 05:02:45.850569 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:45.850486 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ecdc46c-07cd-4c9e-af06-18c6ed1c39f1" containerName="main" Nov 04 05:02:45.850569 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:45.850497 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="029b0ad5-7427-432a-8f07-03c6646a300a" containerName="revoke-keys" Nov 04 05:02:45.854046 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:45.854028 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6d2d59-8hwjz" Nov 04 05:02:45.912321 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:45.911946 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nz4vh\"" Nov 04 05:02:45.912321 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:45.912027 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 05:02:45.912566 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:45.912471 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 05:02:45.918314 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:45.912665 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 05:02:45.918314 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:45.913926 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad6d2d59-8hwjz"] Nov 04 05:02:46.065323 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:46.042071 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98988f22-9f65-416d-b315-95a330fc74ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad6d2d59-8hwjz\" (UID: \"98988f22-9f65-416d-b315-95a330fc74ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6d2d59-8hwjz" Nov 04 05:02:46.065323 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:46.042157 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98988f22-9f65-416d-b315-95a330fc74ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad6d2d59-8hwjz\" (UID: \"98988f22-9f65-416d-b315-95a330fc74ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6d2d59-8hwjz" Nov 04 05:02:46.065323 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:46.042196 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4642d\" (UniqueName: \"kubernetes.io/projected/98988f22-9f65-416d-b315-95a330fc74ad-kube-api-access-4642d\") pod \"maas-api-revoke-sub-ad6d2d59-8hwjz\" (UID: \"98988f22-9f65-416d-b315-95a330fc74ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6d2d59-8hwjz" Nov 04 05:02:46.160365 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:46.148839 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98988f22-9f65-416d-b315-95a330fc74ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad6d2d59-8hwjz\" (UID: \"98988f22-9f65-416d-b315-95a330fc74ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6d2d59-8hwjz" Nov 04 05:02:46.160365 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:46.148938 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98988f22-9f65-416d-b315-95a330fc74ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad6d2d59-8hwjz\" (UID: \"98988f22-9f65-416d-b315-95a330fc74ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6d2d59-8hwjz" Nov 04 05:02:46.160365 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:46.148979 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4642d\" (UniqueName: \"kubernetes.io/projected/98988f22-9f65-416d-b315-95a330fc74ad-kube-api-access-4642d\") pod \"maas-api-revoke-sub-ad6d2d59-8hwjz\" (UID: \"98988f22-9f65-416d-b315-95a330fc74ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6d2d59-8hwjz" Nov 04 05:02:46.160365 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:46.150555 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98988f22-9f65-416d-b315-95a330fc74ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad6d2d59-8hwjz\" (UID: \"98988f22-9f65-416d-b315-95a330fc74ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6d2d59-8hwjz" Nov 04 05:02:46.198322 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:46.194825 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98988f22-9f65-416d-b315-95a330fc74ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad6d2d59-8hwjz\" (UID: \"98988f22-9f65-416d-b315-95a330fc74ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6d2d59-8hwjz" Nov 04 05:02:46.217321 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:46.215212 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4642d\" (UniqueName: \"kubernetes.io/projected/98988f22-9f65-416d-b315-95a330fc74ad-kube-api-access-4642d\") pod \"maas-api-revoke-sub-ad6d2d59-8hwjz\" (UID: \"98988f22-9f65-416d-b315-95a330fc74ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6d2d59-8hwjz" Nov 04 05:02:46.473088 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:46.472990 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6d2d59-8hwjz" Nov 04 05:02:46.858220 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:46.858191 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad6d2d59-8hwjz"] Nov 04 05:02:46.860567 ip-10-0-134-224 kubenswrapper[4395]: W1104 05:02:46.860528 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98988f22_9f65_416d_b315_95a330fc74ad.slice/crio-07eb32b96be91eb7b366ac2318dad57c81d90256a4105e2169a565075a56ba94 WatchSource:0}: Error finding container 07eb32b96be91eb7b366ac2318dad57c81d90256a4105e2169a565075a56ba94: Status 404 returned error can't find the container with id 07eb32b96be91eb7b366ac2318dad57c81d90256a4105e2169a565075a56ba94 Nov 04 05:02:47.289883 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:47.289844 4395 patch_prober.go:29] interesting pod/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 05:02:47.290368 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:47.289898 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" podUID="11dc70f6-5166-4ec0-a746-9d75ee2c6e56" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 05:02:47.718657 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:47.718617 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6d2d59-8hwjz" event={"ID":"98988f22-9f65-416d-b315-95a330fc74ad","Type":"ContainerStarted","Data":"7c1998fa91fd73f13ec4efe3055f0003276c166a76214d0f686a137899ee4505"} Nov 04 05:02:47.718850 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:47.718672 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6d2d59-8hwjz" event={"ID":"98988f22-9f65-416d-b315-95a330fc74ad","Type":"ContainerStarted","Data":"07eb32b96be91eb7b366ac2318dad57c81d90256a4105e2169a565075a56ba94"} Nov 04 05:02:47.828644 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:47.828590 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6d2d59-8hwjz" podStartSLOduration=2.828574541 podStartE2EDuration="2.828574541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:02:47.826702359 +0000 UTC m=+1009.516784807" watchObservedRunningTime="2026-11-04 05:02:47.828574541 +0000 UTC m=+1009.518656988" Nov 04 05:02:48.725848 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:48.725810 4395 generic.go:356] "Generic (PLEG): container finished" podID="98988f22-9f65-416d-b315-95a330fc74ad" containerID="7c1998fa91fd73f13ec4efe3055f0003276c166a76214d0f686a137899ee4505" exitCode=0 Nov 04 05:02:48.726223 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:48.725901 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6d2d59-8hwjz" event={"ID":"98988f22-9f65-416d-b315-95a330fc74ad","Type":"ContainerDied","Data":"7c1998fa91fd73f13ec4efe3055f0003276c166a76214d0f686a137899ee4505"} Nov 04 05:02:49.877890 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:49.877863 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6d2d59-8hwjz" Nov 04 05:02:50.003188 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.003103 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98988f22-9f65-416d-b315-95a330fc74ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98988f22-9f65-416d-b315-95a330fc74ad-maas-api-xks-ca\") pod \"98988f22-9f65-416d-b315-95a330fc74ad\" (UID: \"98988f22-9f65-416d-b315-95a330fc74ad\") " Nov 04 05:02:50.003188 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.003180 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98988f22-9f65-416d-b315-95a330fc74ad-kube-api-access-4642d\" (UniqueName: \"kubernetes.io/projected/98988f22-9f65-416d-b315-95a330fc74ad-kube-api-access-4642d\") pod \"98988f22-9f65-416d-b315-95a330fc74ad\" (UID: \"98988f22-9f65-416d-b315-95a330fc74ad\") " Nov 04 05:02:50.003453 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.003235 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98988f22-9f65-416d-b315-95a330fc74ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98988f22-9f65-416d-b315-95a330fc74ad-maas-api-openshift-ca\") pod \"98988f22-9f65-416d-b315-95a330fc74ad\" (UID: \"98988f22-9f65-416d-b315-95a330fc74ad\") " Nov 04 05:02:50.003721 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.003679 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98988f22-9f65-416d-b315-95a330fc74ad-maas-api-openshift-ca" pod "98988f22-9f65-416d-b315-95a330fc74ad" (UID: "98988f22-9f65-416d-b315-95a330fc74ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:02:50.005274 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.005244 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98988f22-9f65-416d-b315-95a330fc74ad-maas-api-xks-ca" pod "98988f22-9f65-416d-b315-95a330fc74ad" (UID: "98988f22-9f65-416d-b315-95a330fc74ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:02:50.005397 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.005339 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98988f22-9f65-416d-b315-95a330fc74ad-kube-api-access-4642d" pod "98988f22-9f65-416d-b315-95a330fc74ad" (UID: "98988f22-9f65-416d-b315-95a330fc74ad"). InnerVolumeSpecName "kube-api-access-4642d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:02:50.104583 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.104554 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98988f22-9f65-416d-b315-95a330fc74ad-maas-api-xks-ca\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:50.104583 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.104587 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4642d\" (UniqueName: \"kubernetes.io/projected/98988f22-9f65-416d-b315-95a330fc74ad-kube-api-access-4642d\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:50.104759 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.104602 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98988f22-9f65-416d-b315-95a330fc74ad-maas-api-openshift-ca\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:50.117316 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.117270 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-a48f7e/distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p"] Nov 04 05:02:50.122279 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.122256 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-a48f7e/distinct-w4-a48f7e-kserve-8d66794c4-94568"] Nov 04 05:02:50.450877 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.450850 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" Nov 04 05:02:50.488751 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.488725 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" Nov 04 05:02:50.499718 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.499698 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" Nov 04 05:02:50.608380 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.608346 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fed21ce4-c40a-4b31-87cf-bf331f4649b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed21ce4-c40a-4b31-87cf-bf331f4649b8-credential-socket\") pod \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\" (UID: \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\") " Nov 04 05:02:50.608575 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.608428 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-credential-socket\") pod \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\" (UID: \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\") " Nov 04 05:02:50.608575 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.608470 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-istio-podinfo\") pod \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\" (UID: \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\") " Nov 04 05:02:50.608575 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.608496 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3d7a106-096e-4551-819f-c8b166554914-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3d7a106-096e-4551-819f-c8b166554914-workload-certs\") pod \"a3d7a106-096e-4551-819f-c8b166554914\" (UID: \"a3d7a106-096e-4551-819f-c8b166554914\") " Nov 04 05:02:50.608575 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.608556 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fed21ce4-c40a-4b31-87cf-bf331f4649b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fed21ce4-c40a-4b31-87cf-bf331f4649b8-istio-podinfo\") pod \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\" (UID: \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\") " Nov 04 05:02:50.608798 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.608596 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-istio-data\") pod \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\" (UID: \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\") " Nov 04 05:02:50.608798 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.608625 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3d7a106-096e-4551-819f-c8b166554914-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3d7a106-096e-4551-819f-c8b166554914-istio-data\") pod \"a3d7a106-096e-4551-819f-c8b166554914\" (UID: \"a3d7a106-096e-4551-819f-c8b166554914\") " Nov 04 05:02:50.608798 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.608669 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-workload-socket\") pod \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\" (UID: \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\") " Nov 04 05:02:50.608798 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.608688 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fed21ce4-c40a-4b31-87cf-bf331f4649b8-credential-socket" pod "fed21ce4-c40a-4b31-87cf-bf331f4649b8" (UID: "fed21ce4-c40a-4b31-87cf-bf331f4649b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:50.608798 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.608704 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3d7a106-096e-4551-819f-c8b166554914-kube-api-access-ttgsv\" (UniqueName: \"kubernetes.io/projected/a3d7a106-096e-4551-819f-c8b166554914-kube-api-access-ttgsv\") pod \"a3d7a106-096e-4551-819f-c8b166554914\" (UID: \"a3d7a106-096e-4551-819f-c8b166554914\") " Nov 04 05:02:50.608798 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.608745 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3d7a106-096e-4551-819f-c8b166554914-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3d7a106-096e-4551-819f-c8b166554914-istio-envoy\") pod \"a3d7a106-096e-4551-819f-c8b166554914\" (UID: \"a3d7a106-096e-4551-819f-c8b166554914\") " Nov 04 05:02:50.608798 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.608776 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fed21ce4-c40a-4b31-87cf-bf331f4649b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fed21ce4-c40a-4b31-87cf-bf331f4649b8-istio-data\") pod \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\" (UID: \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\") " Nov 04 05:02:50.609152 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.608812 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-istiod-ca-cert\") pod \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\" (UID: \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\") " Nov 04 05:02:50.609152 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.608829 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d7a106-096e-4551-819f-c8b166554914-workload-certs" pod "a3d7a106-096e-4551-819f-c8b166554914" (UID: "a3d7a106-096e-4551-819f-c8b166554914"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:50.609152 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.608850 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3d7a106-096e-4551-819f-c8b166554914-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3d7a106-096e-4551-819f-c8b166554914-istio-podinfo\") pod \"a3d7a106-096e-4551-819f-c8b166554914\" (UID: \"a3d7a106-096e-4551-819f-c8b166554914\") " Nov 04 05:02:50.609152 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.608902 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fed21ce4-c40a-4b31-87cf-bf331f4649b8-kube-api-access-d7wkk\" (UniqueName: \"kubernetes.io/projected/fed21ce4-c40a-4b31-87cf-bf331f4649b8-kube-api-access-d7wkk\") pod \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\" (UID: \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\") " Nov 04 05:02:50.609152 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.608899 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d7a106-096e-4551-819f-c8b166554914-istio-data" pod "a3d7a106-096e-4551-819f-c8b166554914" (UID: "a3d7a106-096e-4551-819f-c8b166554914"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:50.609152 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.608938 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3d7a106-096e-4551-819f-c8b166554914-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3d7a106-096e-4551-819f-c8b166554914-istiod-ca-cert\") pod \"a3d7a106-096e-4551-819f-c8b166554914\" (UID: \"a3d7a106-096e-4551-819f-c8b166554914\") " Nov 04 05:02:50.609152 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.608968 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3d7a106-096e-4551-819f-c8b166554914-istio-token\" (UniqueName: \"kubernetes.io/projected/a3d7a106-096e-4551-819f-c8b166554914-istio-token\") pod \"a3d7a106-096e-4551-819f-c8b166554914\" (UID: \"a3d7a106-096e-4551-819f-c8b166554914\") " Nov 04 05:02:50.609152 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.608999 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-istio-envoy\") pod \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\" (UID: \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\") " Nov 04 05:02:50.609152 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.609046 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-kube-api-access-d58pz\" (UniqueName: \"kubernetes.io/projected/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-kube-api-access-d58pz\") pod \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\" (UID: \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\") " Nov 04 05:02:50.609152 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.609080 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fed21ce4-c40a-4b31-87cf-bf331f4649b8-istio-token\" (UniqueName: \"kubernetes.io/projected/fed21ce4-c40a-4b31-87cf-bf331f4649b8-istio-token\") pod \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\" (UID: \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\") " Nov 04 05:02:50.609152 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.609106 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fed21ce4-c40a-4b31-87cf-bf331f4649b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fed21ce4-c40a-4b31-87cf-bf331f4649b8-istiod-ca-cert\") pod \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\" (UID: \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\") " Nov 04 05:02:50.609152 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.609149 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fed21ce4-c40a-4b31-87cf-bf331f4649b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fed21ce4-c40a-4b31-87cf-bf331f4649b8-istio-envoy\") pod \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\" (UID: \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\") " Nov 04 05:02:50.609744 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.609186 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3d7a106-096e-4551-819f-c8b166554914-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d7a106-096e-4551-819f-c8b166554914-workload-socket\") pod \"a3d7a106-096e-4551-819f-c8b166554914\" (UID: \"a3d7a106-096e-4551-819f-c8b166554914\") " Nov 04 05:02:50.609744 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.609195 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-istiod-ca-cert" pod "11dc70f6-5166-4ec0-a746-9d75ee2c6e56" (UID: "11dc70f6-5166-4ec0-a746-9d75ee2c6e56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:02:50.609744 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.609217 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3d7a106-096e-4551-819f-c8b166554914-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d7a106-096e-4551-819f-c8b166554914-credential-socket\") pod \"a3d7a106-096e-4551-819f-c8b166554914\" (UID: \"a3d7a106-096e-4551-819f-c8b166554914\") " Nov 04 05:02:50.609744 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.609226 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-credential-socket" pod "11dc70f6-5166-4ec0-a746-9d75ee2c6e56" (UID: "11dc70f6-5166-4ec0-a746-9d75ee2c6e56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:50.609744 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.609258 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-istio-data" pod "11dc70f6-5166-4ec0-a746-9d75ee2c6e56" (UID: "11dc70f6-5166-4ec0-a746-9d75ee2c6e56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:50.609744 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.609289 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fed21ce4-c40a-4b31-87cf-bf331f4649b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fed21ce4-c40a-4b31-87cf-bf331f4649b8-workload-certs\") pod \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\" (UID: \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\") " Nov 04 05:02:50.609744 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.609396 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-workload-socket" pod "11dc70f6-5166-4ec0-a746-9d75ee2c6e56" (UID: "11dc70f6-5166-4ec0-a746-9d75ee2c6e56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:50.609744 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.609499 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fed21ce4-c40a-4b31-87cf-bf331f4649b8-istio-data" pod "fed21ce4-c40a-4b31-87cf-bf331f4649b8" (UID: "fed21ce4-c40a-4b31-87cf-bf331f4649b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:50.610207 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.609949 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3d7a106-096e-4551-819f-c8b166554914-istiod-ca-cert" pod "a3d7a106-096e-4551-819f-c8b166554914" (UID: "a3d7a106-096e-4551-819f-c8b166554914"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:02:50.610207 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.610001 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fed21ce4-c40a-4b31-87cf-bf331f4649b8-workload-certs" pod "fed21ce4-c40a-4b31-87cf-bf331f4649b8" (UID: "fed21ce4-c40a-4b31-87cf-bf331f4649b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:50.611400 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.610746 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-workload-certs\") pod \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\" (UID: \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\") " Nov 04 05:02:50.611400 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.610790 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-istio-token\" (UniqueName: \"kubernetes.io/projected/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-istio-token\") pod \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\" (UID: \"11dc70f6-5166-4ec0-a746-9d75ee2c6e56\") " Nov 04 05:02:50.611400 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.610846 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fed21ce4-c40a-4b31-87cf-bf331f4649b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed21ce4-c40a-4b31-87cf-bf331f4649b8-workload-socket\") pod \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\" (UID: \"fed21ce4-c40a-4b31-87cf-bf331f4649b8\") " Nov 04 05:02:50.611400 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.611214 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d7a106-096e-4551-819f-c8b166554914-istio-envoy" pod "a3d7a106-096e-4551-819f-c8b166554914" (UID: "a3d7a106-096e-4551-819f-c8b166554914"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:50.611400 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.611364 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d7a106-096e-4551-819f-c8b166554914-workload-socket" pod "a3d7a106-096e-4551-819f-c8b166554914" (UID: "a3d7a106-096e-4551-819f-c8b166554914"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:50.611729 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.611420 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d7a106-096e-4551-819f-c8b166554914-credential-socket" pod "a3d7a106-096e-4551-819f-c8b166554914" (UID: "a3d7a106-096e-4551-819f-c8b166554914"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:50.611729 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.611469 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fed21ce4-c40a-4b31-87cf-bf331f4649b8-istiod-ca-cert" pod "fed21ce4-c40a-4b31-87cf-bf331f4649b8" (UID: "fed21ce4-c40a-4b31-87cf-bf331f4649b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:02:50.611729 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.611647 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-workload-certs" pod "11dc70f6-5166-4ec0-a746-9d75ee2c6e56" (UID: "11dc70f6-5166-4ec0-a746-9d75ee2c6e56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:50.611897 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.611833 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fed21ce4-c40a-4b31-87cf-bf331f4649b8-workload-socket" pod "fed21ce4-c40a-4b31-87cf-bf331f4649b8" (UID: "fed21ce4-c40a-4b31-87cf-bf331f4649b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:50.611897 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.611884 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-istio-envoy" pod "11dc70f6-5166-4ec0-a746-9d75ee2c6e56" (UID: "11dc70f6-5166-4ec0-a746-9d75ee2c6e56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:50.612005 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.611990 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-istio-data\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:50.612061 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.612010 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3d7a106-096e-4551-819f-c8b166554914-istio-data\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:50.612061 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.612026 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-workload-socket\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:50.612061 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.612043 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3d7a106-096e-4551-819f-c8b166554914-istio-envoy\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:50.612061 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.612057 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fed21ce4-c40a-4b31-87cf-bf331f4649b8-istio-data\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:50.612255 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.612071 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-istiod-ca-cert\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:50.612255 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.612086 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3d7a106-096e-4551-819f-c8b166554914-istiod-ca-cert\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:50.612255 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.612100 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-istio-envoy\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:50.612255 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.612098 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fed21ce4-c40a-4b31-87cf-bf331f4649b8-istio-podinfo" pod "fed21ce4-c40a-4b31-87cf-bf331f4649b8" (UID: "fed21ce4-c40a-4b31-87cf-bf331f4649b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:02:50.612255 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.612114 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fed21ce4-c40a-4b31-87cf-bf331f4649b8-istiod-ca-cert\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:50.612255 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.612168 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d7a106-096e-4551-819f-c8b166554914-workload-socket\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:50.612255 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.612187 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d7a106-096e-4551-819f-c8b166554914-credential-socket\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:50.612255 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.612203 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fed21ce4-c40a-4b31-87cf-bf331f4649b8-workload-certs\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:50.612255 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.612218 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-workload-certs\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:50.612255 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.612230 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed21ce4-c40a-4b31-87cf-bf331f4649b8-workload-socket\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:50.612255 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.612243 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed21ce4-c40a-4b31-87cf-bf331f4649b8-credential-socket\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:50.612255 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.612257 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-credential-socket\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:50.612855 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.612271 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3d7a106-096e-4551-819f-c8b166554914-workload-certs\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:50.612985 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.612949 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-istio-podinfo" pod "11dc70f6-5166-4ec0-a746-9d75ee2c6e56" (UID: "11dc70f6-5166-4ec0-a746-9d75ee2c6e56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:02:50.613186 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.613135 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3d7a106-096e-4551-819f-c8b166554914-istio-podinfo" pod "a3d7a106-096e-4551-819f-c8b166554914" (UID: "a3d7a106-096e-4551-819f-c8b166554914"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:02:50.613341 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.613143 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3d7a106-096e-4551-819f-c8b166554914-kube-api-access-ttgsv" pod "a3d7a106-096e-4551-819f-c8b166554914" (UID: "a3d7a106-096e-4551-819f-c8b166554914"). InnerVolumeSpecName "kube-api-access-ttgsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:02:50.613341 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.613205 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fed21ce4-c40a-4b31-87cf-bf331f4649b8-istio-envoy" pod "fed21ce4-c40a-4b31-87cf-bf331f4649b8" (UID: "fed21ce4-c40a-4b31-87cf-bf331f4649b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:02:50.613341 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.613271 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-kube-api-access-d58pz" pod "11dc70f6-5166-4ec0-a746-9d75ee2c6e56" (UID: "11dc70f6-5166-4ec0-a746-9d75ee2c6e56"). InnerVolumeSpecName "kube-api-access-d58pz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:02:50.614271 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.614229 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fed21ce4-c40a-4b31-87cf-bf331f4649b8-kube-api-access-d7wkk" pod "fed21ce4-c40a-4b31-87cf-bf331f4649b8" (UID: "fed21ce4-c40a-4b31-87cf-bf331f4649b8"). InnerVolumeSpecName "kube-api-access-d7wkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:02:50.614590 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.614565 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3d7a106-096e-4551-819f-c8b166554914-istio-token" pod "a3d7a106-096e-4551-819f-c8b166554914" (UID: "a3d7a106-096e-4551-819f-c8b166554914"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:02:50.614721 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.614704 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fed21ce4-c40a-4b31-87cf-bf331f4649b8-istio-token" pod "fed21ce4-c40a-4b31-87cf-bf331f4649b8" (UID: "fed21ce4-c40a-4b31-87cf-bf331f4649b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:02:50.614783 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.614714 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-istio-token" pod "11dc70f6-5166-4ec0-a746-9d75ee2c6e56" (UID: "11dc70f6-5166-4ec0-a746-9d75ee2c6e56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:02:50.712847 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.712817 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-istio-token\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:50.712847 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.712848 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-istio-podinfo\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:50.713041 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.712860 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fed21ce4-c40a-4b31-87cf-bf331f4649b8-istio-podinfo\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:50.713041 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.712869 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttgsv\" (UniqueName: \"kubernetes.io/projected/a3d7a106-096e-4551-819f-c8b166554914-kube-api-access-ttgsv\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:50.713041 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.712880 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3d7a106-096e-4551-819f-c8b166554914-istio-podinfo\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:50.713041 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.712888 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7wkk\" (UniqueName: \"kubernetes.io/projected/fed21ce4-c40a-4b31-87cf-bf331f4649b8-kube-api-access-d7wkk\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:50.713041 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.712897 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3d7a106-096e-4551-819f-c8b166554914-istio-token\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:50.713041 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.712905 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d58pz\" (UniqueName: \"kubernetes.io/projected/11dc70f6-5166-4ec0-a746-9d75ee2c6e56-kube-api-access-d58pz\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:50.713041 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.712916 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fed21ce4-c40a-4b31-87cf-bf331f4649b8-istio-token\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:50.713041 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.712928 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fed21ce4-c40a-4b31-87cf-bf331f4649b8-istio-envoy\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:02:50.740371 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.740333 4395 generic.go:356] "Generic (PLEG): container finished" podID="11dc70f6-5166-4ec0-a746-9d75ee2c6e56" containerID="1fc2684f299796b9b55ae5a33809495dfe29551cd03fcd9f67195045d9e76417" exitCode=0 Nov 04 05:02:50.740371 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.740363 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" Nov 04 05:02:50.740582 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.740406 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" event={"ID":"11dc70f6-5166-4ec0-a746-9d75ee2c6e56","Type":"ContainerDied","Data":"1fc2684f299796b9b55ae5a33809495dfe29551cd03fcd9f67195045d9e76417"} Nov 04 05:02:50.740582 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.740484 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r" event={"ID":"11dc70f6-5166-4ec0-a746-9d75ee2c6e56","Type":"ContainerDied","Data":"43f72e8956bf6fb36559366274e8cc5457c82d92447053ae52b5d56e5e126b81"} Nov 04 05:02:50.740582 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.740519 4395 scope.go:122] "RemoveContainer" containerID="1fc2684f299796b9b55ae5a33809495dfe29551cd03fcd9f67195045d9e76417" Nov 04 05:02:50.742031 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.742010 4395 generic.go:356] "Generic (PLEG): container finished" podID="fed21ce4-c40a-4b31-87cf-bf331f4649b8" containerID="b4a06fc077e7beb6572b9601846a5ab3e98fb6d4c850b028842a738fa79f4b56" exitCode=0 Nov 04 05:02:50.742174 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.742055 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" Nov 04 05:02:50.742174 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.742083 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" event={"ID":"fed21ce4-c40a-4b31-87cf-bf331f4649b8","Type":"ContainerDied","Data":"b4a06fc077e7beb6572b9601846a5ab3e98fb6d4c850b028842a738fa79f4b56"} Nov 04 05:02:50.742174 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.742138 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n" event={"ID":"fed21ce4-c40a-4b31-87cf-bf331f4649b8","Type":"ContainerDied","Data":"028230f753e72da514efd123cf46a9ccf361254871b12f9a7f942c54394268fe"} Nov 04 05:02:50.749503 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.749179 4395 generic.go:356] "Generic (PLEG): container finished" podID="a3d7a106-096e-4551-819f-c8b166554914" containerID="2bbbefde8fe524095f1c456d56cdb7dbc0abb470b208f02d14cea8700557cfac" exitCode=0 Nov 04 05:02:50.749503 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.749240 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" Nov 04 05:02:50.749503 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.749267 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" event={"ID":"a3d7a106-096e-4551-819f-c8b166554914","Type":"ContainerDied","Data":"2bbbefde8fe524095f1c456d56cdb7dbc0abb470b208f02d14cea8700557cfac"} Nov 04 05:02:50.749503 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.749346 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5" event={"ID":"a3d7a106-096e-4551-819f-c8b166554914","Type":"ContainerDied","Data":"9250642e5642d7a0a2b79172e04c5beada54c53e3f6273888d95bad9eb184463"} Nov 04 05:02:50.758274 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.758200 4395 scope.go:122] "RemoveContainer" containerID="1fc2684f299796b9b55ae5a33809495dfe29551cd03fcd9f67195045d9e76417" Nov 04 05:02:50.759578 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:02:50.759431 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fc2684f299796b9b55ae5a33809495dfe29551cd03fcd9f67195045d9e76417\": container with ID starting with 1fc2684f299796b9b55ae5a33809495dfe29551cd03fcd9f67195045d9e76417 not found: ID does not exist" containerID="1fc2684f299796b9b55ae5a33809495dfe29551cd03fcd9f67195045d9e76417" Nov 04 05:02:50.759578 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.759472 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fc2684f299796b9b55ae5a33809495dfe29551cd03fcd9f67195045d9e76417"} err="failed to get container status \"1fc2684f299796b9b55ae5a33809495dfe29551cd03fcd9f67195045d9e76417\": rpc error: code = NotFound desc = could not find container \"1fc2684f299796b9b55ae5a33809495dfe29551cd03fcd9f67195045d9e76417\": container with ID starting with 1fc2684f299796b9b55ae5a33809495dfe29551cd03fcd9f67195045d9e76417 not found: ID does not exist" Nov 04 05:02:50.759578 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.759496 4395 scope.go:122] "RemoveContainer" containerID="b4a06fc077e7beb6572b9601846a5ab3e98fb6d4c850b028842a738fa79f4b56" Nov 04 05:02:50.760438 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.760417 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6d2d59-8hwjz" Nov 04 05:02:50.760528 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.760445 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6d2d59-8hwjz" event={"ID":"98988f22-9f65-416d-b315-95a330fc74ad","Type":"ContainerDied","Data":"07eb32b96be91eb7b366ac2318dad57c81d90256a4105e2169a565075a56ba94"} Nov 04 05:02:50.760528 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.760478 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07eb32b96be91eb7b366ac2318dad57c81d90256a4105e2169a565075a56ba94" Nov 04 05:02:50.768616 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.768599 4395 scope.go:122] "RemoveContainer" containerID="b4a06fc077e7beb6572b9601846a5ab3e98fb6d4c850b028842a738fa79f4b56" Nov 04 05:02:50.768855 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:02:50.768838 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4a06fc077e7beb6572b9601846a5ab3e98fb6d4c850b028842a738fa79f4b56\": container with ID starting with b4a06fc077e7beb6572b9601846a5ab3e98fb6d4c850b028842a738fa79f4b56 not found: ID does not exist" containerID="b4a06fc077e7beb6572b9601846a5ab3e98fb6d4c850b028842a738fa79f4b56" Nov 04 05:02:50.768893 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.768862 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4a06fc077e7beb6572b9601846a5ab3e98fb6d4c850b028842a738fa79f4b56"} err="failed to get container status \"b4a06fc077e7beb6572b9601846a5ab3e98fb6d4c850b028842a738fa79f4b56\": rpc error: code = NotFound desc = could not find container \"b4a06fc077e7beb6572b9601846a5ab3e98fb6d4c850b028842a738fa79f4b56\": container with ID starting with b4a06fc077e7beb6572b9601846a5ab3e98fb6d4c850b028842a738fa79f4b56 not found: ID does not exist" Nov 04 05:02:50.768893 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.768878 4395 scope.go:122] "RemoveContainer" containerID="2bbbefde8fe524095f1c456d56cdb7dbc0abb470b208f02d14cea8700557cfac" Nov 04 05:02:50.777350 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.777328 4395 scope.go:122] "RemoveContainer" containerID="2bbbefde8fe524095f1c456d56cdb7dbc0abb470b208f02d14cea8700557cfac" Nov 04 05:02:50.777611 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:02:50.777593 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bbbefde8fe524095f1c456d56cdb7dbc0abb470b208f02d14cea8700557cfac\": container with ID starting with 2bbbefde8fe524095f1c456d56cdb7dbc0abb470b208f02d14cea8700557cfac not found: ID does not exist" containerID="2bbbefde8fe524095f1c456d56cdb7dbc0abb470b208f02d14cea8700557cfac" Nov 04 05:02:50.777674 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.777618 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bbbefde8fe524095f1c456d56cdb7dbc0abb470b208f02d14cea8700557cfac"} err="failed to get container status \"2bbbefde8fe524095f1c456d56cdb7dbc0abb470b208f02d14cea8700557cfac\": rpc error: code = NotFound desc = could not find container \"2bbbefde8fe524095f1c456d56cdb7dbc0abb470b208f02d14cea8700557cfac\": container with ID starting with 2bbbefde8fe524095f1c456d56cdb7dbc0abb470b208f02d14cea8700557cfac not found: ID does not exist" Nov 04 05:02:50.894565 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.894541 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n"] Nov 04 05:02:50.900063 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:50.900042 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-qzh6n"] Nov 04 05:02:51.051405 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:51.051377 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r"] Nov 04 05:02:51.055737 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:51.055712 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-lhh6r"] Nov 04 05:02:51.158194 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:51.158116 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5"] Nov 04 05:02:51.161912 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:51.161887 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c5bb6d-openshift-default-64f56dfd77-69pz5"] Nov 04 05:02:52.415980 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:52.415948 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11dc70f6-5166-4ec0-a746-9d75ee2c6e56" path="/var/lib/kubelet/pods/11dc70f6-5166-4ec0-a746-9d75ee2c6e56/volumes" Nov 04 05:02:52.416410 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:52.416394 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3d7a106-096e-4551-819f-c8b166554914" path="/var/lib/kubelet/pods/a3d7a106-096e-4551-819f-c8b166554914/volumes" Nov 04 05:02:52.416787 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:52.416776 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fed21ce4-c40a-4b31-87cf-bf331f4649b8" path="/var/lib/kubelet/pods/fed21ce4-c40a-4b31-87cf-bf331f4649b8/volumes" Nov 04 05:02:56.916144 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:56.916103 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg"] Nov 04 05:02:56.916592 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:56.916543 4395 cpu_manager.go:414] "containerMap: removing container" podUID="11dc70f6-5166-4ec0-a746-9d75ee2c6e56" containerName="istio-proxy" Nov 04 05:02:56.916592 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:56.916559 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11dc70f6-5166-4ec0-a746-9d75ee2c6e56" containerName="istio-proxy" Nov 04 05:02:56.916592 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:56.916573 4395 cpu_manager.go:414] "containerMap: removing container" podUID="fed21ce4-c40a-4b31-87cf-bf331f4649b8" containerName="istio-proxy" Nov 04 05:02:56.916592 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:56.916578 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fed21ce4-c40a-4b31-87cf-bf331f4649b8" containerName="istio-proxy" Nov 04 05:02:56.916737 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:56.916601 4395 cpu_manager.go:414] "containerMap: removing container" podUID="a3d7a106-096e-4551-819f-c8b166554914" containerName="istio-proxy" Nov 04 05:02:56.916737 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:56.916606 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3d7a106-096e-4551-819f-c8b166554914" containerName="istio-proxy" Nov 04 05:02:56.916737 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:56.916620 4395 cpu_manager.go:414] "containerMap: removing container" podUID="98988f22-9f65-416d-b315-95a330fc74ad" containerName="revoke-keys" Nov 04 05:02:56.916737 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:56.916628 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98988f22-9f65-416d-b315-95a330fc74ad" containerName="revoke-keys" Nov 04 05:02:56.916737 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:56.916706 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="98988f22-9f65-416d-b315-95a330fc74ad" containerName="revoke-keys" Nov 04 05:02:56.916737 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:56.916718 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="11dc70f6-5166-4ec0-a746-9d75ee2c6e56" containerName="istio-proxy" Nov 04 05:02:56.916737 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:56.916724 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="fed21ce4-c40a-4b31-87cf-bf331f4649b8" containerName="istio-proxy" Nov 04 05:02:56.916737 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:56.916733 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3d7a106-096e-4551-819f-c8b166554914" containerName="istio-proxy" Nov 04 05:02:56.922026 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:56.922008 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" Nov 04 05:02:57.019645 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:57.019606 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg"] Nov 04 05:02:57.076807 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:57.076774 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-istio-data\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg\" (UID: \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" Nov 04 05:02:57.076807 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:57.076811 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-istiod-ca-cert\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg\" (UID: \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" Nov 04 05:02:57.077012 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:57.076937 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-istio-podinfo\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg\" (UID: \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" Nov 04 05:02:57.077012 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:57.076975 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-credential-socket\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg\" (UID: \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" Nov 04 05:02:57.077082 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:57.077056 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-workload-socket\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg\" (UID: \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" Nov 04 05:02:57.077116 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:57.077097 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-istio-token\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg\" (UID: \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" Nov 04 05:02:57.077146 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:57.077121 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-workload-certs\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg\" (UID: \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" Nov 04 05:02:57.077188 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:57.077175 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-istio-envoy\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg\" (UID: \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" Nov 04 05:02:57.077224 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:57.077201 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skgg5\" (UniqueName: \"kubernetes.io/projected/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-kube-api-access-skgg5\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg\" (UID: \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" Nov 04 05:02:57.178229 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:57.178155 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-workload-certs\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg\" (UID: \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" Nov 04 05:02:57.178229 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:57.178202 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-istio-envoy\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg\" (UID: \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" Nov 04 05:02:57.178229 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:57.178223 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skgg5\" (UniqueName: \"kubernetes.io/projected/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-kube-api-access-skgg5\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg\" (UID: \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" Nov 04 05:02:57.178531 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:57.178249 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-istio-data\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg\" (UID: \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" Nov 04 05:02:57.178531 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:57.178270 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-istiod-ca-cert\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg\" (UID: \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" Nov 04 05:02:57.178531 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:57.178337 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-istio-podinfo\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg\" (UID: \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" Nov 04 05:02:57.178531 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:57.178372 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-credential-socket\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg\" (UID: \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" Nov 04 05:02:57.178531 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:57.178428 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-workload-socket\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg\" (UID: \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" Nov 04 05:02:57.178531 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:57.178457 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-istio-token\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg\" (UID: \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" Nov 04 05:02:57.178842 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:57.178615 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-workload-certs\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg\" (UID: \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" Nov 04 05:02:57.178842 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:57.178702 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-istio-data\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg\" (UID: \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" Nov 04 05:02:57.178962 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:57.178870 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-credential-socket\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg\" (UID: \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" Nov 04 05:02:57.178962 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:57.178908 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-workload-socket\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg\" (UID: \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" Nov 04 05:02:57.179110 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:57.179088 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-istiod-ca-cert\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg\" (UID: \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" Nov 04 05:02:57.180519 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:57.180500 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-istio-envoy\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg\" (UID: \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" Nov 04 05:02:57.180744 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:57.180726 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-istio-podinfo\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg\" (UID: \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" Nov 04 05:02:57.235518 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:57.235485 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-istio-token\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg\" (UID: \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" Nov 04 05:02:57.235691 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:57.235654 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skgg5\" (UniqueName: \"kubernetes.io/projected/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-kube-api-access-skgg5\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg\" (UID: \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" Nov 04 05:02:57.535043 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:57.534969 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" Nov 04 05:02:57.993928 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:57.993878 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg"] Nov 04 05:02:57.995164 ip-10-0-134-224 kubenswrapper[4395]: W1104 05:02:57.995137 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9cd31cb_c46f_4e16_b289_d2546b1e6d9c.slice/crio-e09ecdc4ca97bbb09c19d4d3a2b76ac9ebd14ea21e801a94f8f4e8ff54f7c7b0 WatchSource:0}: Error finding container e09ecdc4ca97bbb09c19d4d3a2b76ac9ebd14ea21e801a94f8f4e8ff54f7c7b0: Status 404 returned error can't find the container with id e09ecdc4ca97bbb09c19d4d3a2b76ac9ebd14ea21e801a94f8f4e8ff54f7c7b0 Nov 04 05:02:57.997480 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:57.997440 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 05:02:57.997601 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:57.997522 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 05:02:57.997601 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:57.997567 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 05:02:58.681323 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:58.680703 4395 scope.go:122] "RemoveContainer" containerID="106d718d23426bfbae521a1071b7b9bffd115cf74f6a1b71cae79af7f101192e" Nov 04 05:02:58.812803 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:58.812765 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" event={"ID":"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c","Type":"ContainerStarted","Data":"971faaa13fc0e1ddaeeee871578543ecce8edaf920044995631d7dbb13b4894a"} Nov 04 05:02:58.812978 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:58.812821 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" event={"ID":"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c","Type":"ContainerStarted","Data":"e09ecdc4ca97bbb09c19d4d3a2b76ac9ebd14ea21e801a94f8f4e8ff54f7c7b0"} Nov 04 05:02:58.984747 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:58.984645 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" podStartSLOduration=2.984629793 podStartE2EDuration="2.984629793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:02:58.973330911 +0000 UTC m=+1020.663413362" watchObservedRunningTime="2026-11-04 05:02:58.984629793 +0000 UTC m=+1020.674712240" Nov 04 05:02:59.535147 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:59.535115 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" Nov 04 05:02:59.536622 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:59.536593 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 04 05:02:59.536760 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:02:59.536649 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" podUID="b9cd31cb-c46f-4e16-b289-d2546b1e6d9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 04 05:03:00.535582 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:00.535551 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 04 05:03:00.535970 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:00.535601 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" podUID="b9cd31cb-c46f-4e16-b289-d2546b1e6d9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 04 05:03:00.899321 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:00.898972 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g"] Nov 04 05:03:00.899321 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:00.899227 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" podUID="54c856ac-40f6-4c13-8ae5-8282967b7496" containerName="istio-proxy" containerID="cri-o://0fa8f8564f87f678aee07f25dad5c0d576a45af16729581f2c02447c491eeff9" gracePeriod=30 Nov 04 05:03:00.913315 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:00.905489 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7"] Nov 04 05:03:00.913315 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:00.905764 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" podUID="7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3" containerName="istio-proxy" containerID="cri-o://b891c8d18324b1bcf09778beb94367415dc1db0632fad3296f15789f7af7cc44" gracePeriod=30 Nov 04 05:03:01.535781 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:01.535747 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 04 05:03:01.536154 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:01.535803 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" podUID="b9cd31cb-c46f-4e16-b289-d2546b1e6d9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 04 05:03:02.114320 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:02.111809 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e8560ec-8l27l"] Nov 04 05:03:02.121346 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:02.118500 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8560ec-8l27l" Nov 04 05:03:02.139902 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:02.139865 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x87cd\" (UniqueName: \"kubernetes.io/projected/4739a60d-77e7-4164-a229-ffd5122ce861-kube-api-access-x87cd\") pod \"maas-api-revoke-sub-5e8560ec-8l27l\" (UID: \"4739a60d-77e7-4164-a229-ffd5122ce861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8560ec-8l27l" Nov 04 05:03:02.140069 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:02.139909 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4739a60d-77e7-4164-a229-ffd5122ce861-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e8560ec-8l27l\" (UID: \"4739a60d-77e7-4164-a229-ffd5122ce861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8560ec-8l27l" Nov 04 05:03:02.140069 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:02.139995 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4739a60d-77e7-4164-a229-ffd5122ce861-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e8560ec-8l27l\" (UID: \"4739a60d-77e7-4164-a229-ffd5122ce861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8560ec-8l27l" Nov 04 05:03:02.172496 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:02.172457 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 05:03:02.218088 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:02.218049 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nz4vh\"" Nov 04 05:03:02.218286 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:02.218168 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 05:03:02.218286 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:02.218259 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 05:03:02.221055 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:02.221020 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e8560ec-8l27l"] Nov 04 05:03:02.241049 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:02.241015 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4739a60d-77e7-4164-a229-ffd5122ce861-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e8560ec-8l27l\" (UID: \"4739a60d-77e7-4164-a229-ffd5122ce861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8560ec-8l27l" Nov 04 05:03:02.241207 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:02.241082 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4739a60d-77e7-4164-a229-ffd5122ce861-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e8560ec-8l27l\" (UID: \"4739a60d-77e7-4164-a229-ffd5122ce861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8560ec-8l27l" Nov 04 05:03:02.241266 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:02.241221 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x87cd\" (UniqueName: \"kubernetes.io/projected/4739a60d-77e7-4164-a229-ffd5122ce861-kube-api-access-x87cd\") pod \"maas-api-revoke-sub-5e8560ec-8l27l\" (UID: \"4739a60d-77e7-4164-a229-ffd5122ce861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8560ec-8l27l" Nov 04 05:03:02.242175 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:02.242150 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4739a60d-77e7-4164-a229-ffd5122ce861-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e8560ec-8l27l\" (UID: \"4739a60d-77e7-4164-a229-ffd5122ce861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8560ec-8l27l" Nov 04 05:03:02.243692 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:02.243666 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4739a60d-77e7-4164-a229-ffd5122ce861-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e8560ec-8l27l\" (UID: \"4739a60d-77e7-4164-a229-ffd5122ce861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8560ec-8l27l" Nov 04 05:03:02.298795 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:02.298766 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x87cd\" (UniqueName: \"kubernetes.io/projected/4739a60d-77e7-4164-a229-ffd5122ce861-kube-api-access-x87cd\") pod \"maas-api-revoke-sub-5e8560ec-8l27l\" (UID: \"4739a60d-77e7-4164-a229-ffd5122ce861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8560ec-8l27l" Nov 04 05:03:02.434030 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:02.433950 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8560ec-8l27l" Nov 04 05:03:02.537673 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:02.537642 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 04 05:03:02.537976 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:02.537709 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" podUID="b9cd31cb-c46f-4e16-b289-d2546b1e6d9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 04 05:03:02.846760 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:02.846736 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e8560ec-8l27l"] Nov 04 05:03:02.847962 ip-10-0-134-224 kubenswrapper[4395]: W1104 05:03:02.847931 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4739a60d_77e7_4164_a229_ffd5122ce861.slice/crio-c661b257ca6117381d3a659239ea11bb4f9349cfddcee4b1d20442e986253c75 WatchSource:0}: Error finding container c661b257ca6117381d3a659239ea11bb4f9349cfddcee4b1d20442e986253c75: Status 404 returned error can't find the container with id c661b257ca6117381d3a659239ea11bb4f9349cfddcee4b1d20442e986253c75 Nov 04 05:03:03.485554 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:03:03.485467 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 05:03:03.536127 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:03.536093 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 04 05:03:03.536350 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:03.536323 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" podUID="b9cd31cb-c46f-4e16-b289-d2546b1e6d9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 04 05:03:03.845919 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:03.845885 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8560ec-8l27l" event={"ID":"4739a60d-77e7-4164-a229-ffd5122ce861","Type":"ContainerStarted","Data":"bd4d503038d3a0807a6e85a2ac638220afcad2d945e08088ddb3f559d45c63dd"} Nov 04 05:03:03.846405 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:03.845937 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8560ec-8l27l" event={"ID":"4739a60d-77e7-4164-a229-ffd5122ce861","Type":"ContainerStarted","Data":"c661b257ca6117381d3a659239ea11bb4f9349cfddcee4b1d20442e986253c75"} Nov 04 05:03:03.955527 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:03.955483 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8560ec-8l27l" podStartSLOduration=1.95546894 podStartE2EDuration="1.95546894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:03:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:03:03.953243356 +0000 UTC m=+1025.643325803" watchObservedRunningTime="2026-11-04 05:03:03.95546894 +0000 UTC m=+1025.645551437" Nov 04 05:03:04.536086 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:04.536053 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 04 05:03:04.536238 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:04.536115 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" podUID="b9cd31cb-c46f-4e16-b289-d2546b1e6d9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 04 05:03:04.852391 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:04.852350 4395 generic.go:356] "Generic (PLEG): container finished" podID="4739a60d-77e7-4164-a229-ffd5122ce861" containerID="bd4d503038d3a0807a6e85a2ac638220afcad2d945e08088ddb3f559d45c63dd" exitCode=0 Nov 04 05:03:04.852763 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:04.852434 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8560ec-8l27l" event={"ID":"4739a60d-77e7-4164-a229-ffd5122ce861","Type":"ContainerDied","Data":"bd4d503038d3a0807a6e85a2ac638220afcad2d945e08088ddb3f559d45c63dd"} Nov 04 05:03:05.136973 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.136888 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-a48f7e/distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p" podUID="124ecb9d-bcbe-415a-9ba7-5789af9afb98" containerName="main" containerID="cri-o://b4e6be4e2be9679058cb9cdf79fd824472db7f46011fac4a9b09e98eb7c583a9" gracePeriod=45 Nov 04 05:03:05.141497 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.141455 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-a48f7e/distinct-w4-a48f7e-kserve-8d66794c4-94568" podUID="61c310e9-1d98-4656-9105-107acf3d4669" containerName="main" containerID="cri-o://f4d7570710b9fc665411cb9855e692c2434f1f8cd16b2da2e650978d0bd06873" gracePeriod=45 Nov 04 05:03:05.208210 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.208176 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 05:03:05.208365 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.208237 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" podUID="7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 05:03:05.420391 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.420364 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a48f7e/distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p" Nov 04 05:03:05.439761 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.439739 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a48f7e/distinct-w4-a48f7e-kserve-8d66794c4-94568" Nov 04 05:03:05.496663 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.496638 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/124ecb9d-bcbe-415a-9ba7-5789af9afb98-dshm\" (UniqueName: \"kubernetes.io/empty-dir/124ecb9d-bcbe-415a-9ba7-5789af9afb98-dshm\") pod \"124ecb9d-bcbe-415a-9ba7-5789af9afb98\" (UID: \"124ecb9d-bcbe-415a-9ba7-5789af9afb98\") " Nov 04 05:03:05.496811 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.496687 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c310e9-1d98-4656-9105-107acf3d4669-dshm\" (UniqueName: \"kubernetes.io/empty-dir/61c310e9-1d98-4656-9105-107acf3d4669-dshm\") pod \"61c310e9-1d98-4656-9105-107acf3d4669\" (UID: \"61c310e9-1d98-4656-9105-107acf3d4669\") " Nov 04 05:03:05.496811 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.496706 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/124ecb9d-bcbe-415a-9ba7-5789af9afb98-tls-certs\" (UniqueName: \"kubernetes.io/secret/124ecb9d-bcbe-415a-9ba7-5789af9afb98-tls-certs\") pod \"124ecb9d-bcbe-415a-9ba7-5789af9afb98\" (UID: \"124ecb9d-bcbe-415a-9ba7-5789af9afb98\") " Nov 04 05:03:05.496811 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.496750 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c310e9-1d98-4656-9105-107acf3d4669-home\" (UniqueName: \"kubernetes.io/empty-dir/61c310e9-1d98-4656-9105-107acf3d4669-home\") pod \"61c310e9-1d98-4656-9105-107acf3d4669\" (UID: \"61c310e9-1d98-4656-9105-107acf3d4669\") " Nov 04 05:03:05.496811 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.496771 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61c310e9-1d98-4656-9105-107acf3d4669-tls-certs\" (UniqueName: \"kubernetes.io/secret/61c310e9-1d98-4656-9105-107acf3d4669-tls-certs\") pod \"61c310e9-1d98-4656-9105-107acf3d4669\" (UID: \"61c310e9-1d98-4656-9105-107acf3d4669\") " Nov 04 05:03:05.496811 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.496790 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/124ecb9d-bcbe-415a-9ba7-5789af9afb98-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124ecb9d-bcbe-415a-9ba7-5789af9afb98-model-cache\") pod \"124ecb9d-bcbe-415a-9ba7-5789af9afb98\" (UID: \"124ecb9d-bcbe-415a-9ba7-5789af9afb98\") " Nov 04 05:03:05.497069 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.496852 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61c310e9-1d98-4656-9105-107acf3d4669-kube-api-access-l9hqc\" (UniqueName: \"kubernetes.io/projected/61c310e9-1d98-4656-9105-107acf3d4669-kube-api-access-l9hqc\") pod \"61c310e9-1d98-4656-9105-107acf3d4669\" (UID: \"61c310e9-1d98-4656-9105-107acf3d4669\") " Nov 04 05:03:05.497069 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.496885 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/124ecb9d-bcbe-415a-9ba7-5789af9afb98-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124ecb9d-bcbe-415a-9ba7-5789af9afb98-tmp-dir\") pod \"124ecb9d-bcbe-415a-9ba7-5789af9afb98\" (UID: \"124ecb9d-bcbe-415a-9ba7-5789af9afb98\") " Nov 04 05:03:05.497069 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.496925 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/124ecb9d-bcbe-415a-9ba7-5789af9afb98-kube-api-access-pbjn2\" (UniqueName: \"kubernetes.io/projected/124ecb9d-bcbe-415a-9ba7-5789af9afb98-kube-api-access-pbjn2\") pod \"124ecb9d-bcbe-415a-9ba7-5789af9afb98\" (UID: \"124ecb9d-bcbe-415a-9ba7-5789af9afb98\") " Nov 04 05:03:05.497069 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.496954 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/124ecb9d-bcbe-415a-9ba7-5789af9afb98-home\" (UniqueName: \"kubernetes.io/empty-dir/124ecb9d-bcbe-415a-9ba7-5789af9afb98-home\") pod \"124ecb9d-bcbe-415a-9ba7-5789af9afb98\" (UID: \"124ecb9d-bcbe-415a-9ba7-5789af9afb98\") " Nov 04 05:03:05.497069 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.496996 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c310e9-1d98-4656-9105-107acf3d4669-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61c310e9-1d98-4656-9105-107acf3d4669-tmp-dir\") pod \"61c310e9-1d98-4656-9105-107acf3d4669\" (UID: \"61c310e9-1d98-4656-9105-107acf3d4669\") " Nov 04 05:03:05.497069 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.497043 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c310e9-1d98-4656-9105-107acf3d4669-home" pod "61c310e9-1d98-4656-9105-107acf3d4669" (UID: "61c310e9-1d98-4656-9105-107acf3d4669"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:03:05.497069 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.497059 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c310e9-1d98-4656-9105-107acf3d4669-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61c310e9-1d98-4656-9105-107acf3d4669-model-cache\") pod \"61c310e9-1d98-4656-9105-107acf3d4669\" (UID: \"61c310e9-1d98-4656-9105-107acf3d4669\") " Nov 04 05:03:05.497501 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.497479 4395 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61c310e9-1d98-4656-9105-107acf3d4669-home\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:05.497741 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.497614 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124ecb9d-bcbe-415a-9ba7-5789af9afb98-model-cache" pod "124ecb9d-bcbe-415a-9ba7-5789af9afb98" (UID: "124ecb9d-bcbe-415a-9ba7-5789af9afb98"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:03:05.497741 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.497699 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c310e9-1d98-4656-9105-107acf3d4669-model-cache" pod "61c310e9-1d98-4656-9105-107acf3d4669" (UID: "61c310e9-1d98-4656-9105-107acf3d4669"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:03:05.497938 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.497888 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c310e9-1d98-4656-9105-107acf3d4669-tmp-dir" pod "61c310e9-1d98-4656-9105-107acf3d4669" (UID: "61c310e9-1d98-4656-9105-107acf3d4669"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:03:05.498007 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.497955 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124ecb9d-bcbe-415a-9ba7-5789af9afb98-home" pod "124ecb9d-bcbe-415a-9ba7-5789af9afb98" (UID: "124ecb9d-bcbe-415a-9ba7-5789af9afb98"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:03:05.498007 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.497963 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124ecb9d-bcbe-415a-9ba7-5789af9afb98-tmp-dir" pod "124ecb9d-bcbe-415a-9ba7-5789af9afb98" (UID: "124ecb9d-bcbe-415a-9ba7-5789af9afb98"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:03:05.499445 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.499416 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/124ecb9d-bcbe-415a-9ba7-5789af9afb98-kube-api-access-pbjn2" pod "124ecb9d-bcbe-415a-9ba7-5789af9afb98" (UID: "124ecb9d-bcbe-415a-9ba7-5789af9afb98"). InnerVolumeSpecName "kube-api-access-pbjn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:03:05.499537 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.499424 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/124ecb9d-bcbe-415a-9ba7-5789af9afb98-tls-certs" pod "124ecb9d-bcbe-415a-9ba7-5789af9afb98" (UID: "124ecb9d-bcbe-415a-9ba7-5789af9afb98"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:03:05.499537 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.499458 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c310e9-1d98-4656-9105-107acf3d4669-dshm" pod "61c310e9-1d98-4656-9105-107acf3d4669" (UID: "61c310e9-1d98-4656-9105-107acf3d4669"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:03:05.499625 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.499572 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124ecb9d-bcbe-415a-9ba7-5789af9afb98-dshm" pod "124ecb9d-bcbe-415a-9ba7-5789af9afb98" (UID: "124ecb9d-bcbe-415a-9ba7-5789af9afb98"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:03:05.499936 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.499911 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61c310e9-1d98-4656-9105-107acf3d4669-kube-api-access-l9hqc" pod "61c310e9-1d98-4656-9105-107acf3d4669" (UID: "61c310e9-1d98-4656-9105-107acf3d4669"). InnerVolumeSpecName "kube-api-access-l9hqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:03:05.500017 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.499991 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61c310e9-1d98-4656-9105-107acf3d4669-tls-certs" pod "61c310e9-1d98-4656-9105-107acf3d4669" (UID: "61c310e9-1d98-4656-9105-107acf3d4669"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:03:05.598424 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.598391 4395 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61c310e9-1d98-4656-9105-107acf3d4669-model-cache\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:05.598424 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.598419 4395 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/124ecb9d-bcbe-415a-9ba7-5789af9afb98-dshm\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:05.598424 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.598429 4395 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61c310e9-1d98-4656-9105-107acf3d4669-dshm\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:05.598659 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.598438 4395 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/124ecb9d-bcbe-415a-9ba7-5789af9afb98-tls-certs\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:05.598659 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.598447 4395 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61c310e9-1d98-4656-9105-107acf3d4669-tls-certs\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:05.598659 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.598455 4395 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124ecb9d-bcbe-415a-9ba7-5789af9afb98-model-cache\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:05.598659 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.598465 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9hqc\" (UniqueName: \"kubernetes.io/projected/61c310e9-1d98-4656-9105-107acf3d4669-kube-api-access-l9hqc\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:05.598659 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.598473 4395 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124ecb9d-bcbe-415a-9ba7-5789af9afb98-tmp-dir\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:05.598659 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.598482 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbjn2\" (UniqueName: \"kubernetes.io/projected/124ecb9d-bcbe-415a-9ba7-5789af9afb98-kube-api-access-pbjn2\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:05.598659 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.598490 4395 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/124ecb9d-bcbe-415a-9ba7-5789af9afb98-home\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:05.598659 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.598497 4395 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61c310e9-1d98-4656-9105-107acf3d4669-tmp-dir\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:05.711415 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.711336 4395 patch_prober.go:29] interesting pod/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Nov 04 05:03:05.711415 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.711388 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" podUID="54c856ac-40f6-4c13-8ae5-8282967b7496" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 05:03:05.861352 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.861323 4395 generic.go:356] "Generic (PLEG): container finished" podID="124ecb9d-bcbe-415a-9ba7-5789af9afb98" containerID="b4e6be4e2be9679058cb9cdf79fd824472db7f46011fac4a9b09e98eb7c583a9" exitCode=0 Nov 04 05:03:05.861795 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.861402 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a48f7e/distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p" event={"ID":"124ecb9d-bcbe-415a-9ba7-5789af9afb98","Type":"ContainerDied","Data":"b4e6be4e2be9679058cb9cdf79fd824472db7f46011fac4a9b09e98eb7c583a9"} Nov 04 05:03:05.861795 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.861449 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a48f7e/distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p" Nov 04 05:03:05.861795 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.861457 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a48f7e/distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p" event={"ID":"124ecb9d-bcbe-415a-9ba7-5789af9afb98","Type":"ContainerDied","Data":"2e7025449bef8e4a0348af47844d7b366af6c20c0556834d3465528bb03ec81f"} Nov 04 05:03:05.861795 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.861480 4395 scope.go:122] "RemoveContainer" containerID="b4e6be4e2be9679058cb9cdf79fd824472db7f46011fac4a9b09e98eb7c583a9" Nov 04 05:03:05.862941 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.862919 4395 generic.go:356] "Generic (PLEG): container finished" podID="61c310e9-1d98-4656-9105-107acf3d4669" containerID="f4d7570710b9fc665411cb9855e692c2434f1f8cd16b2da2e650978d0bd06873" exitCode=0 Nov 04 05:03:05.863056 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.863032 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a48f7e/distinct-w4-a48f7e-kserve-8d66794c4-94568" Nov 04 05:03:05.863135 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.863061 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a48f7e/distinct-w4-a48f7e-kserve-8d66794c4-94568" event={"ID":"61c310e9-1d98-4656-9105-107acf3d4669","Type":"ContainerDied","Data":"f4d7570710b9fc665411cb9855e692c2434f1f8cd16b2da2e650978d0bd06873"} Nov 04 05:03:05.863135 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.863106 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a48f7e/distinct-w4-a48f7e-kserve-8d66794c4-94568" event={"ID":"61c310e9-1d98-4656-9105-107acf3d4669","Type":"ContainerDied","Data":"84579e818ae393393683fb3a2d2fefd5223c67abdf20f484ac5ae9303b330989"} Nov 04 05:03:05.872605 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.872584 4395 scope.go:122] "RemoveContainer" containerID="b4e6be4e2be9679058cb9cdf79fd824472db7f46011fac4a9b09e98eb7c583a9" Nov 04 05:03:05.872895 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:03:05.872868 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4e6be4e2be9679058cb9cdf79fd824472db7f46011fac4a9b09e98eb7c583a9\": container with ID starting with b4e6be4e2be9679058cb9cdf79fd824472db7f46011fac4a9b09e98eb7c583a9 not found: ID does not exist" containerID="b4e6be4e2be9679058cb9cdf79fd824472db7f46011fac4a9b09e98eb7c583a9" Nov 04 05:03:05.872987 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.872904 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4e6be4e2be9679058cb9cdf79fd824472db7f46011fac4a9b09e98eb7c583a9"} err="failed to get container status \"b4e6be4e2be9679058cb9cdf79fd824472db7f46011fac4a9b09e98eb7c583a9\": rpc error: code = NotFound desc = could not find container \"b4e6be4e2be9679058cb9cdf79fd824472db7f46011fac4a9b09e98eb7c583a9\": container with ID starting with b4e6be4e2be9679058cb9cdf79fd824472db7f46011fac4a9b09e98eb7c583a9 not found: ID does not exist" Nov 04 05:03:05.872987 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.872927 4395 scope.go:122] "RemoveContainer" containerID="f4d7570710b9fc665411cb9855e692c2434f1f8cd16b2da2e650978d0bd06873" Nov 04 05:03:05.884500 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.884483 4395 scope.go:122] "RemoveContainer" containerID="f4d7570710b9fc665411cb9855e692c2434f1f8cd16b2da2e650978d0bd06873" Nov 04 05:03:05.884775 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:03:05.884753 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4d7570710b9fc665411cb9855e692c2434f1f8cd16b2da2e650978d0bd06873\": container with ID starting with f4d7570710b9fc665411cb9855e692c2434f1f8cd16b2da2e650978d0bd06873 not found: ID does not exist" containerID="f4d7570710b9fc665411cb9855e692c2434f1f8cd16b2da2e650978d0bd06873" Nov 04 05:03:05.884836 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:05.884785 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4d7570710b9fc665411cb9855e692c2434f1f8cd16b2da2e650978d0bd06873"} err="failed to get container status \"f4d7570710b9fc665411cb9855e692c2434f1f8cd16b2da2e650978d0bd06873\": rpc error: code = NotFound desc = could not find container \"f4d7570710b9fc665411cb9855e692c2434f1f8cd16b2da2e650978d0bd06873\": container with ID starting with f4d7570710b9fc665411cb9855e692c2434f1f8cd16b2da2e650978d0bd06873 not found: ID does not exist" Nov 04 05:03:06.022058 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.022033 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-a48f7e/distinct-w4-a48f7e-kserve-8d66794c4-94568"] Nov 04 05:03:06.025919 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.025894 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-a48f7e/distinct-w4-a48f7e-kserve-8d66794c4-94568"] Nov 04 05:03:06.034084 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.034064 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8560ec-8l27l" Nov 04 05:03:06.105082 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.104957 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4739a60d-77e7-4164-a229-ffd5122ce861-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4739a60d-77e7-4164-a229-ffd5122ce861-maas-api-xks-ca\") pod \"4739a60d-77e7-4164-a229-ffd5122ce861\" (UID: \"4739a60d-77e7-4164-a229-ffd5122ce861\") " Nov 04 05:03:06.105082 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.105026 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4739a60d-77e7-4164-a229-ffd5122ce861-kube-api-access-x87cd\" (UniqueName: \"kubernetes.io/projected/4739a60d-77e7-4164-a229-ffd5122ce861-kube-api-access-x87cd\") pod \"4739a60d-77e7-4164-a229-ffd5122ce861\" (UID: \"4739a60d-77e7-4164-a229-ffd5122ce861\") " Nov 04 05:03:06.106464 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.105785 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4739a60d-77e7-4164-a229-ffd5122ce861-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4739a60d-77e7-4164-a229-ffd5122ce861-maas-api-openshift-ca\") pod \"4739a60d-77e7-4164-a229-ffd5122ce861\" (UID: \"4739a60d-77e7-4164-a229-ffd5122ce861\") " Nov 04 05:03:06.107063 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.107018 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4739a60d-77e7-4164-a229-ffd5122ce861-maas-api-openshift-ca" pod "4739a60d-77e7-4164-a229-ffd5122ce861" (UID: "4739a60d-77e7-4164-a229-ffd5122ce861"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:03:06.114228 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.114181 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4739a60d-77e7-4164-a229-ffd5122ce861-maas-api-xks-ca" pod "4739a60d-77e7-4164-a229-ffd5122ce861" (UID: "4739a60d-77e7-4164-a229-ffd5122ce861"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 05:03:06.115410 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.115368 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4739a60d-77e7-4164-a229-ffd5122ce861-kube-api-access-x87cd" pod "4739a60d-77e7-4164-a229-ffd5122ce861" (UID: "4739a60d-77e7-4164-a229-ffd5122ce861"). InnerVolumeSpecName "kube-api-access-x87cd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:03:06.127211 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.127186 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-a48f7e/distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p"] Nov 04 05:03:06.132340 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.132321 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-a48f7e/distinct-2-w4-a48f7e-kserve-c95c5d4bf-q524p"] Nov 04 05:03:06.187356 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.187332 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" Nov 04 05:03:06.209702 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.207980 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4739a60d-77e7-4164-a229-ffd5122ce861-maas-api-xks-ca\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:06.209702 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.208034 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x87cd\" (UniqueName: \"kubernetes.io/projected/4739a60d-77e7-4164-a229-ffd5122ce861-kube-api-access-x87cd\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:06.209702 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.208081 4395 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4739a60d-77e7-4164-a229-ffd5122ce861-maas-api-openshift-ca\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:06.242408 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.242369 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" Nov 04 05:03:06.308684 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.308656 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-istio-data\") pod \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\" (UID: \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\") " Nov 04 05:03:06.308848 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.308698 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54c856ac-40f6-4c13-8ae5-8282967b7496-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c856ac-40f6-4c13-8ae5-8282967b7496-istio-data\") pod \"54c856ac-40f6-4c13-8ae5-8282967b7496\" (UID: \"54c856ac-40f6-4c13-8ae5-8282967b7496\") " Nov 04 05:03:06.308848 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.308716 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54c856ac-40f6-4c13-8ae5-8282967b7496-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c856ac-40f6-4c13-8ae5-8282967b7496-credential-socket\") pod \"54c856ac-40f6-4c13-8ae5-8282967b7496\" (UID: \"54c856ac-40f6-4c13-8ae5-8282967b7496\") " Nov 04 05:03:06.308848 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.308742 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-workload-certs\") pod \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\" (UID: \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\") " Nov 04 05:03:06.308848 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.308771 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54c856ac-40f6-4c13-8ae5-8282967b7496-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c856ac-40f6-4c13-8ae5-8282967b7496-istiod-ca-cert\") pod \"54c856ac-40f6-4c13-8ae5-8282967b7496\" (UID: \"54c856ac-40f6-4c13-8ae5-8282967b7496\") " Nov 04 05:03:06.308848 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.308812 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54c856ac-40f6-4c13-8ae5-8282967b7496-istio-token\" (UniqueName: \"kubernetes.io/projected/54c856ac-40f6-4c13-8ae5-8282967b7496-istio-token\") pod \"54c856ac-40f6-4c13-8ae5-8282967b7496\" (UID: \"54c856ac-40f6-4c13-8ae5-8282967b7496\") " Nov 04 05:03:06.308848 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.308846 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-istio-podinfo\") pod \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\" (UID: \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\") " Nov 04 05:03:06.309182 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.308874 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54c856ac-40f6-4c13-8ae5-8282967b7496-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c856ac-40f6-4c13-8ae5-8282967b7496-istio-envoy\") pod \"54c856ac-40f6-4c13-8ae5-8282967b7496\" (UID: \"54c856ac-40f6-4c13-8ae5-8282967b7496\") " Nov 04 05:03:06.309182 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.308903 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-credential-socket\") pod \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\" (UID: \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\") " Nov 04 05:03:06.309182 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.308940 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-workload-socket\") pod \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\" (UID: \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\") " Nov 04 05:03:06.309182 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.309006 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54c856ac-40f6-4c13-8ae5-8282967b7496-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c856ac-40f6-4c13-8ae5-8282967b7496-workload-socket\") pod \"54c856ac-40f6-4c13-8ae5-8282967b7496\" (UID: \"54c856ac-40f6-4c13-8ae5-8282967b7496\") " Nov 04 05:03:06.309182 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.309067 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54c856ac-40f6-4c13-8ae5-8282967b7496-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c856ac-40f6-4c13-8ae5-8282967b7496-istio-podinfo\") pod \"54c856ac-40f6-4c13-8ae5-8282967b7496\" (UID: \"54c856ac-40f6-4c13-8ae5-8282967b7496\") " Nov 04 05:03:06.309182 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.309113 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-istiod-ca-cert\") pod \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\" (UID: \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\") " Nov 04 05:03:06.309182 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.309161 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54c856ac-40f6-4c13-8ae5-8282967b7496-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c856ac-40f6-4c13-8ae5-8282967b7496-workload-certs\") pod \"54c856ac-40f6-4c13-8ae5-8282967b7496\" (UID: \"54c856ac-40f6-4c13-8ae5-8282967b7496\") " Nov 04 05:03:06.309564 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.309183 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54c856ac-40f6-4c13-8ae5-8282967b7496-istiod-ca-cert" pod "54c856ac-40f6-4c13-8ae5-8282967b7496" (UID: "54c856ac-40f6-4c13-8ae5-8282967b7496"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:03:06.309564 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.309204 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-kube-api-access-scrtb\" (UniqueName: \"kubernetes.io/projected/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-kube-api-access-scrtb\") pod \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\" (UID: \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\") " Nov 04 05:03:06.309564 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.309236 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-istio-token\" (UniqueName: \"kubernetes.io/projected/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-istio-token\") pod \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\" (UID: \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\") " Nov 04 05:03:06.309564 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.309273 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-istio-envoy\") pod \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\" (UID: \"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3\") " Nov 04 05:03:06.309564 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.309345 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54c856ac-40f6-4c13-8ae5-8282967b7496-kube-api-access-pl7gj\" (UniqueName: \"kubernetes.io/projected/54c856ac-40f6-4c13-8ae5-8282967b7496-kube-api-access-pl7gj\") pod \"54c856ac-40f6-4c13-8ae5-8282967b7496\" (UID: \"54c856ac-40f6-4c13-8ae5-8282967b7496\") " Nov 04 05:03:06.309564 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.309478 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c856ac-40f6-4c13-8ae5-8282967b7496-istio-data" pod "54c856ac-40f6-4c13-8ae5-8282967b7496" (UID: "54c856ac-40f6-4c13-8ae5-8282967b7496"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:03:06.309888 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.309714 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c856ac-40f6-4c13-8ae5-8282967b7496-credential-socket" pod "54c856ac-40f6-4c13-8ae5-8282967b7496" (UID: "54c856ac-40f6-4c13-8ae5-8282967b7496"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:03:06.309888 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.309750 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c856ac-40f6-4c13-8ae5-8282967b7496-istio-data\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:06.309888 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.309773 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c856ac-40f6-4c13-8ae5-8282967b7496-istiod-ca-cert\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:06.310037 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.309947 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-workload-certs" pod "7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3" (UID: "7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:03:06.310134 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.310110 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c856ac-40f6-4c13-8ae5-8282967b7496-workload-certs" pod "54c856ac-40f6-4c13-8ae5-8282967b7496" (UID: "54c856ac-40f6-4c13-8ae5-8282967b7496"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:03:06.310623 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.310460 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-istiod-ca-cert" pod "7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3" (UID: "7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:03:06.310772 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.310746 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c856ac-40f6-4c13-8ae5-8282967b7496-workload-socket" pod "54c856ac-40f6-4c13-8ae5-8282967b7496" (UID: "54c856ac-40f6-4c13-8ae5-8282967b7496"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:03:06.311385 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.311347 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-workload-socket" pod "7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3" (UID: "7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:03:06.311767 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.311745 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-credential-socket" pod "7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3" (UID: "7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:03:06.311989 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.311792 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-istio-data" pod "7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3" (UID: "7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:03:06.313045 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.313021 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c856ac-40f6-4c13-8ae5-8282967b7496-kube-api-access-pl7gj" pod "54c856ac-40f6-4c13-8ae5-8282967b7496" (UID: "54c856ac-40f6-4c13-8ae5-8282967b7496"). InnerVolumeSpecName "kube-api-access-pl7gj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:03:06.315998 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.315967 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c856ac-40f6-4c13-8ae5-8282967b7496-istio-envoy" pod "54c856ac-40f6-4c13-8ae5-8282967b7496" (UID: "54c856ac-40f6-4c13-8ae5-8282967b7496"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:03:06.315998 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.315972 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-istio-token" pod "7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3" (UID: "7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:03:06.316246 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.316217 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-kube-api-access-scrtb" pod "7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3" (UID: "7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3"). InnerVolumeSpecName "kube-api-access-scrtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:03:06.316471 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.316445 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54c856ac-40f6-4c13-8ae5-8282967b7496-istio-podinfo" pod "54c856ac-40f6-4c13-8ae5-8282967b7496" (UID: "54c856ac-40f6-4c13-8ae5-8282967b7496"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:03:06.316539 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.316527 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c856ac-40f6-4c13-8ae5-8282967b7496-istio-token" pod "54c856ac-40f6-4c13-8ae5-8282967b7496" (UID: "54c856ac-40f6-4c13-8ae5-8282967b7496"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:03:06.316845 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.316826 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-istio-podinfo" pod "7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3" (UID: "7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:03:06.318255 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.318236 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-istio-envoy" pod "7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3" (UID: "7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:03:06.410947 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.410909 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-istio-token\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:06.410947 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.410945 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-istio-envoy\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:06.411164 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.410962 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl7gj\" (UniqueName: \"kubernetes.io/projected/54c856ac-40f6-4c13-8ae5-8282967b7496-kube-api-access-pl7gj\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:06.411164 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.410979 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-istio-data\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:06.411164 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.410993 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c856ac-40f6-4c13-8ae5-8282967b7496-credential-socket\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:06.411164 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.411005 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-workload-certs\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:06.411164 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.411018 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54c856ac-40f6-4c13-8ae5-8282967b7496-istio-token\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:06.411164 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.411030 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-istio-podinfo\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:06.411164 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.411042 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c856ac-40f6-4c13-8ae5-8282967b7496-istio-envoy\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:06.411164 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.411054 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-credential-socket\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:06.411164 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.411068 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-workload-socket\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:06.411164 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.411081 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c856ac-40f6-4c13-8ae5-8282967b7496-workload-socket\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:06.411164 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.411092 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c856ac-40f6-4c13-8ae5-8282967b7496-istio-podinfo\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:06.411164 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.411104 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-istiod-ca-cert\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:06.411164 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.411116 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c856ac-40f6-4c13-8ae5-8282967b7496-workload-certs\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:06.411164 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.411128 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scrtb\" (UniqueName: \"kubernetes.io/projected/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3-kube-api-access-scrtb\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:06.416886 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.416856 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="124ecb9d-bcbe-415a-9ba7-5789af9afb98" path="/var/lib/kubelet/pods/124ecb9d-bcbe-415a-9ba7-5789af9afb98/volumes" Nov 04 05:03:06.417255 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.417242 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61c310e9-1d98-4656-9105-107acf3d4669" path="/var/lib/kubelet/pods/61c310e9-1d98-4656-9105-107acf3d4669/volumes" Nov 04 05:03:06.577493 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.577449 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 05:03:06.577803 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.577753 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" podUID="b9cd31cb-c46f-4e16-b289-d2546b1e6d9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 05:03:06.874459 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.874346 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8560ec-8l27l" event={"ID":"4739a60d-77e7-4164-a229-ffd5122ce861","Type":"ContainerDied","Data":"c661b257ca6117381d3a659239ea11bb4f9349cfddcee4b1d20442e986253c75"} Nov 04 05:03:06.874459 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.874417 4395 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c661b257ca6117381d3a659239ea11bb4f9349cfddcee4b1d20442e986253c75" Nov 04 05:03:06.874459 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.874370 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8560ec-8l27l" Nov 04 05:03:06.877865 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.877811 4395 generic.go:356] "Generic (PLEG): container finished" podID="7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3" containerID="b891c8d18324b1bcf09778beb94367415dc1db0632fad3296f15789f7af7cc44" exitCode=0 Nov 04 05:03:06.877865 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.877857 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" Nov 04 05:03:06.878086 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.877965 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" event={"ID":"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3","Type":"ContainerDied","Data":"b891c8d18324b1bcf09778beb94367415dc1db0632fad3296f15789f7af7cc44"} Nov 04 05:03:06.878147 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.878122 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7" event={"ID":"7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3","Type":"ContainerDied","Data":"f02ee307fd68d360dad77af5981bdda36587a2999dea67354fbec78e7a487869"} Nov 04 05:03:06.878204 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.878172 4395 scope.go:122] "RemoveContainer" containerID="b891c8d18324b1bcf09778beb94367415dc1db0632fad3296f15789f7af7cc44" Nov 04 05:03:06.880606 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.880574 4395 generic.go:356] "Generic (PLEG): container finished" podID="54c856ac-40f6-4c13-8ae5-8282967b7496" containerID="0fa8f8564f87f678aee07f25dad5c0d576a45af16729581f2c02447c491eeff9" exitCode=0 Nov 04 05:03:06.880756 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.880746 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" Nov 04 05:03:06.880850 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.880739 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" event={"ID":"54c856ac-40f6-4c13-8ae5-8282967b7496","Type":"ContainerDied","Data":"0fa8f8564f87f678aee07f25dad5c0d576a45af16729581f2c02447c491eeff9"} Nov 04 05:03:06.880921 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.880902 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g" event={"ID":"54c856ac-40f6-4c13-8ae5-8282967b7496","Type":"ContainerDied","Data":"fabe6da8a84f88dd3b2e737d5145ce493ce41164d42933f0a6582e285e7ed839"} Nov 04 05:03:06.889408 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.889386 4395 scope.go:122] "RemoveContainer" containerID="96dc40bd86256d67a635da6966dd828d2c6f96b60a30da6fd1ff64b4a5e007a0" Nov 04 05:03:06.898904 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.898882 4395 scope.go:122] "RemoveContainer" containerID="b891c8d18324b1bcf09778beb94367415dc1db0632fad3296f15789f7af7cc44" Nov 04 05:03:06.899202 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:03:06.899183 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b891c8d18324b1bcf09778beb94367415dc1db0632fad3296f15789f7af7cc44\": container with ID starting with b891c8d18324b1bcf09778beb94367415dc1db0632fad3296f15789f7af7cc44 not found: ID does not exist" containerID="b891c8d18324b1bcf09778beb94367415dc1db0632fad3296f15789f7af7cc44" Nov 04 05:03:06.899272 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.899212 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b891c8d18324b1bcf09778beb94367415dc1db0632fad3296f15789f7af7cc44"} err="failed to get container status \"b891c8d18324b1bcf09778beb94367415dc1db0632fad3296f15789f7af7cc44\": rpc error: code = NotFound desc = could not find container \"b891c8d18324b1bcf09778beb94367415dc1db0632fad3296f15789f7af7cc44\": container with ID starting with b891c8d18324b1bcf09778beb94367415dc1db0632fad3296f15789f7af7cc44 not found: ID does not exist" Nov 04 05:03:06.899272 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.899228 4395 scope.go:122] "RemoveContainer" containerID="96dc40bd86256d67a635da6966dd828d2c6f96b60a30da6fd1ff64b4a5e007a0" Nov 04 05:03:06.899571 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:03:06.899546 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96dc40bd86256d67a635da6966dd828d2c6f96b60a30da6fd1ff64b4a5e007a0\": container with ID starting with 96dc40bd86256d67a635da6966dd828d2c6f96b60a30da6fd1ff64b4a5e007a0 not found: ID does not exist" containerID="96dc40bd86256d67a635da6966dd828d2c6f96b60a30da6fd1ff64b4a5e007a0" Nov 04 05:03:06.899616 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.899576 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96dc40bd86256d67a635da6966dd828d2c6f96b60a30da6fd1ff64b4a5e007a0"} err="failed to get container status \"96dc40bd86256d67a635da6966dd828d2c6f96b60a30da6fd1ff64b4a5e007a0\": rpc error: code = NotFound desc = could not find container \"96dc40bd86256d67a635da6966dd828d2c6f96b60a30da6fd1ff64b4a5e007a0\": container with ID starting with 96dc40bd86256d67a635da6966dd828d2c6f96b60a30da6fd1ff64b4a5e007a0 not found: ID does not exist" Nov 04 05:03:06.899616 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.899591 4395 scope.go:122] "RemoveContainer" containerID="0fa8f8564f87f678aee07f25dad5c0d576a45af16729581f2c02447c491eeff9" Nov 04 05:03:06.912339 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.912313 4395 scope.go:122] "RemoveContainer" containerID="0fa8f8564f87f678aee07f25dad5c0d576a45af16729581f2c02447c491eeff9" Nov 04 05:03:06.912723 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:03:06.912696 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fa8f8564f87f678aee07f25dad5c0d576a45af16729581f2c02447c491eeff9\": container with ID starting with 0fa8f8564f87f678aee07f25dad5c0d576a45af16729581f2c02447c491eeff9 not found: ID does not exist" containerID="0fa8f8564f87f678aee07f25dad5c0d576a45af16729581f2c02447c491eeff9" Nov 04 05:03:06.912847 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.912729 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fa8f8564f87f678aee07f25dad5c0d576a45af16729581f2c02447c491eeff9"} err="failed to get container status \"0fa8f8564f87f678aee07f25dad5c0d576a45af16729581f2c02447c491eeff9\": rpc error: code = NotFound desc = could not find container \"0fa8f8564f87f678aee07f25dad5c0d576a45af16729581f2c02447c491eeff9\": container with ID starting with 0fa8f8564f87f678aee07f25dad5c0d576a45af16729581f2c02447c491eeff9 not found: ID does not exist" Nov 04 05:03:06.933025 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.933000 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" Nov 04 05:03:06.933334 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.933311 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" Nov 04 05:03:06.933961 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:06.933944 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" Nov 04 05:03:07.085256 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:07.085224 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7"] Nov 04 05:03:07.090419 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:07.090391 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-jn5w7"] Nov 04 05:03:07.218503 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:07.218466 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g"] Nov 04 05:03:07.221379 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:07.221351 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-204e38-openshift-default-6d488f5d7f-7wq5g"] Nov 04 05:03:08.417707 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:08.417673 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54c856ac-40f6-4c13-8ae5-8282967b7496" path="/var/lib/kubelet/pods/54c856ac-40f6-4c13-8ae5-8282967b7496/volumes" Nov 04 05:03:08.418119 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:08.418105 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3" path="/var/lib/kubelet/pods/7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3/volumes" Nov 04 05:03:12.028354 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.028253 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q"] Nov 04 05:03:12.028874 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.028852 4395 cpu_manager.go:414] "containerMap: removing container" podUID="61c310e9-1d98-4656-9105-107acf3d4669" containerName="main" Nov 04 05:03:12.028931 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.028879 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61c310e9-1d98-4656-9105-107acf3d4669" containerName="main" Nov 04 05:03:12.028931 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.028896 4395 cpu_manager.go:414] "containerMap: removing container" podUID="7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3" containerName="istio-proxy" Nov 04 05:03:12.028931 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.028903 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3" containerName="istio-proxy" Nov 04 05:03:12.028931 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.028916 4395 cpu_manager.go:414] "containerMap: removing container" podUID="4739a60d-77e7-4164-a229-ffd5122ce861" containerName="revoke-keys" Nov 04 05:03:12.028931 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.028924 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4739a60d-77e7-4164-a229-ffd5122ce861" containerName="revoke-keys" Nov 04 05:03:12.029135 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.028945 4395 cpu_manager.go:414] "containerMap: removing container" podUID="7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3" containerName="istio-proxy" Nov 04 05:03:12.029135 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.028953 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3" containerName="istio-proxy" Nov 04 05:03:12.029135 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.028962 4395 cpu_manager.go:414] "containerMap: removing container" podUID="54c856ac-40f6-4c13-8ae5-8282967b7496" containerName="istio-proxy" Nov 04 05:03:12.029135 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.028969 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54c856ac-40f6-4c13-8ae5-8282967b7496" containerName="istio-proxy" Nov 04 05:03:12.029135 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.029008 4395 cpu_manager.go:414] "containerMap: removing container" podUID="124ecb9d-bcbe-415a-9ba7-5789af9afb98" containerName="main" Nov 04 05:03:12.029135 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.029015 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="124ecb9d-bcbe-415a-9ba7-5789af9afb98" containerName="main" Nov 04 05:03:12.029135 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.029112 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="61c310e9-1d98-4656-9105-107acf3d4669" containerName="main" Nov 04 05:03:12.029135 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.029122 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3" containerName="istio-proxy" Nov 04 05:03:12.029135 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.029131 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="4739a60d-77e7-4164-a229-ffd5122ce861" containerName="revoke-keys" Nov 04 05:03:12.029484 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.029141 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e150c0f-44ce-4946-ac17-bbf8ac9bfbb3" containerName="istio-proxy" Nov 04 05:03:12.029484 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.029149 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="54c856ac-40f6-4c13-8ae5-8282967b7496" containerName="istio-proxy" Nov 04 05:03:12.029484 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.029163 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="124ecb9d-bcbe-415a-9ba7-5789af9afb98" containerName="main" Nov 04 05:03:12.034379 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.034355 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" Nov 04 05:03:12.074153 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.074122 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2f924d7-f797-4ed5-b57d-bb2271751fed-istio-podinfo\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q\" (UID: \"e2f924d7-f797-4ed5-b57d-bb2271751fed\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" Nov 04 05:03:12.074153 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.074157 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2f924d7-f797-4ed5-b57d-bb2271751fed-istio-data\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q\" (UID: \"e2f924d7-f797-4ed5-b57d-bb2271751fed\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" Nov 04 05:03:12.074356 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.074180 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2f924d7-f797-4ed5-b57d-bb2271751fed-istio-envoy\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q\" (UID: \"e2f924d7-f797-4ed5-b57d-bb2271751fed\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" Nov 04 05:03:12.074393 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.074356 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2f924d7-f797-4ed5-b57d-bb2271751fed-workload-certs\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q\" (UID: \"e2f924d7-f797-4ed5-b57d-bb2271751fed\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" Nov 04 05:03:12.074430 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.074398 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2f924d7-f797-4ed5-b57d-bb2271751fed-istio-token\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q\" (UID: \"e2f924d7-f797-4ed5-b57d-bb2271751fed\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" Nov 04 05:03:12.074430 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.074420 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpbvb\" (UniqueName: \"kubernetes.io/projected/e2f924d7-f797-4ed5-b57d-bb2271751fed-kube-api-access-bpbvb\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q\" (UID: \"e2f924d7-f797-4ed5-b57d-bb2271751fed\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" Nov 04 05:03:12.074487 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.074458 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2f924d7-f797-4ed5-b57d-bb2271751fed-istiod-ca-cert\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q\" (UID: \"e2f924d7-f797-4ed5-b57d-bb2271751fed\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" Nov 04 05:03:12.074600 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.074578 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f924d7-f797-4ed5-b57d-bb2271751fed-credential-socket\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q\" (UID: \"e2f924d7-f797-4ed5-b57d-bb2271751fed\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" Nov 04 05:03:12.074657 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.074639 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f924d7-f797-4ed5-b57d-bb2271751fed-workload-socket\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q\" (UID: \"e2f924d7-f797-4ed5-b57d-bb2271751fed\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" Nov 04 05:03:12.135932 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.135899 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q"] Nov 04 05:03:12.177770 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.177731 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpbvb\" (UniqueName: \"kubernetes.io/projected/e2f924d7-f797-4ed5-b57d-bb2271751fed-kube-api-access-bpbvb\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q\" (UID: \"e2f924d7-f797-4ed5-b57d-bb2271751fed\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" Nov 04 05:03:12.177955 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.177776 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2f924d7-f797-4ed5-b57d-bb2271751fed-istiod-ca-cert\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q\" (UID: \"e2f924d7-f797-4ed5-b57d-bb2271751fed\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" Nov 04 05:03:12.177955 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.177814 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f924d7-f797-4ed5-b57d-bb2271751fed-credential-socket\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q\" (UID: \"e2f924d7-f797-4ed5-b57d-bb2271751fed\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" Nov 04 05:03:12.177955 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.177854 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f924d7-f797-4ed5-b57d-bb2271751fed-workload-socket\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q\" (UID: \"e2f924d7-f797-4ed5-b57d-bb2271751fed\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" Nov 04 05:03:12.177955 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.177932 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2f924d7-f797-4ed5-b57d-bb2271751fed-istio-podinfo\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q\" (UID: \"e2f924d7-f797-4ed5-b57d-bb2271751fed\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" Nov 04 05:03:12.178194 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.177960 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2f924d7-f797-4ed5-b57d-bb2271751fed-istio-data\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q\" (UID: \"e2f924d7-f797-4ed5-b57d-bb2271751fed\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" Nov 04 05:03:12.178194 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.177999 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2f924d7-f797-4ed5-b57d-bb2271751fed-istio-envoy\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q\" (UID: \"e2f924d7-f797-4ed5-b57d-bb2271751fed\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" Nov 04 05:03:12.178194 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.178058 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2f924d7-f797-4ed5-b57d-bb2271751fed-workload-certs\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q\" (UID: \"e2f924d7-f797-4ed5-b57d-bb2271751fed\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" Nov 04 05:03:12.178194 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.178103 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2f924d7-f797-4ed5-b57d-bb2271751fed-istio-token\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q\" (UID: \"e2f924d7-f797-4ed5-b57d-bb2271751fed\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" Nov 04 05:03:12.178419 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.178279 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f924d7-f797-4ed5-b57d-bb2271751fed-workload-socket\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q\" (UID: \"e2f924d7-f797-4ed5-b57d-bb2271751fed\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" Nov 04 05:03:12.178419 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.178366 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2f924d7-f797-4ed5-b57d-bb2271751fed-istio-data\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q\" (UID: \"e2f924d7-f797-4ed5-b57d-bb2271751fed\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" Nov 04 05:03:12.178638 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.178616 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2f924d7-f797-4ed5-b57d-bb2271751fed-istiod-ca-cert\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q\" (UID: \"e2f924d7-f797-4ed5-b57d-bb2271751fed\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" Nov 04 05:03:12.178733 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.178649 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2f924d7-f797-4ed5-b57d-bb2271751fed-workload-certs\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q\" (UID: \"e2f924d7-f797-4ed5-b57d-bb2271751fed\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" Nov 04 05:03:12.178786 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.178727 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f924d7-f797-4ed5-b57d-bb2271751fed-credential-socket\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q\" (UID: \"e2f924d7-f797-4ed5-b57d-bb2271751fed\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" Nov 04 05:03:12.180495 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.180468 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2f924d7-f797-4ed5-b57d-bb2271751fed-istio-envoy\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q\" (UID: \"e2f924d7-f797-4ed5-b57d-bb2271751fed\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" Nov 04 05:03:12.180627 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.180603 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2f924d7-f797-4ed5-b57d-bb2271751fed-istio-podinfo\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q\" (UID: \"e2f924d7-f797-4ed5-b57d-bb2271751fed\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" Nov 04 05:03:12.234750 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.234716 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2f924d7-f797-4ed5-b57d-bb2271751fed-istio-token\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q\" (UID: \"e2f924d7-f797-4ed5-b57d-bb2271751fed\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" Nov 04 05:03:12.234892 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.234841 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpbvb\" (UniqueName: \"kubernetes.io/projected/e2f924d7-f797-4ed5-b57d-bb2271751fed-kube-api-access-bpbvb\") pod \"e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q\" (UID: \"e2f924d7-f797-4ed5-b57d-bb2271751fed\") " pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" Nov 04 05:03:12.348365 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.348332 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" Nov 04 05:03:12.777946 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.777923 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q"] Nov 04 05:03:12.780859 ip-10-0-134-224 kubenswrapper[4395]: W1104 05:03:12.780826 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2f924d7_f797_4ed5_b57d_bb2271751fed.slice/crio-e45b7d7f0c490bfeff9dcabd152a8efe1dfcd66accb064ab94963f7186cbf5a0 WatchSource:0}: Error finding container e45b7d7f0c490bfeff9dcabd152a8efe1dfcd66accb064ab94963f7186cbf5a0: Status 404 returned error can't find the container with id e45b7d7f0c490bfeff9dcabd152a8efe1dfcd66accb064ab94963f7186cbf5a0 Nov 04 05:03:12.783080 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.783039 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 05:03:12.783151 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.783134 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 05:03:12.783195 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.783178 4395 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 05:03:12.940827 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.940796 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" event={"ID":"e2f924d7-f797-4ed5-b57d-bb2271751fed","Type":"ContainerStarted","Data":"7b44bd1edc430eece9f1fab445d13ad78d2f633b65b5f692e438e206688c9819"} Nov 04 05:03:12.940967 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:12.940850 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" event={"ID":"e2f924d7-f797-4ed5-b57d-bb2271751fed","Type":"ContainerStarted","Data":"e45b7d7f0c490bfeff9dcabd152a8efe1dfcd66accb064ab94963f7186cbf5a0"} Nov 04 05:03:13.055284 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:13.055230 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" podStartSLOduration=2.055214662 podStartE2EDuration="2.055214662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 05:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:03:13.052950639 +0000 UTC m=+1034.743033086" watchObservedRunningTime="2026-11-04 05:03:13.055214662 +0000 UTC m=+1034.745297109" Nov 04 05:03:13.348981 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:13.348905 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" Nov 04 05:03:13.350360 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:13.350339 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 04 05:03:13.350422 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:13.350385 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" podUID="e2f924d7-f797-4ed5-b57d-bb2271751fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 04 05:03:14.349534 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:14.349497 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 04 05:03:14.349903 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:14.349554 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" podUID="e2f924d7-f797-4ed5-b57d-bb2271751fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 04 05:03:15.349595 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:15.349558 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 04 05:03:15.349958 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:15.349610 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" podUID="e2f924d7-f797-4ed5-b57d-bb2271751fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 04 05:03:16.349258 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:16.349218 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 04 05:03:16.349547 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:16.349275 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" podUID="e2f924d7-f797-4ed5-b57d-bb2271751fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 04 05:03:17.349101 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:17.349069 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 04 05:03:17.349512 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:17.349126 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" podUID="e2f924d7-f797-4ed5-b57d-bb2271751fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 04 05:03:18.349153 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:18.349119 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 04 05:03:18.349579 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:18.349171 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" podUID="e2f924d7-f797-4ed5-b57d-bb2271751fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 04 05:03:19.348767 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:19.348732 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 04 05:03:19.348942 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:19.348779 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" podUID="e2f924d7-f797-4ed5-b57d-bb2271751fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 04 05:03:21.390466 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:21.390429 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 05:03:21.390935 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:21.390489 4395 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" podUID="e2f924d7-f797-4ed5-b57d-bb2271751fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 05:03:21.821038 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:21.821003 4395 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" Nov 04 05:03:21.821492 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:21.821469 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" Nov 04 05:03:21.822611 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:21.822590 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" Nov 04 05:03:31.759706 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:31.759669 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj"] Nov 04 05:03:31.760156 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:31.759929 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" podUID="5798fb27-7552-42f7-ae48-c4ff7decff83" containerName="istio-proxy" containerID="cri-o://f689453b02d1e5b4f217640b7db32540b25105755d4f44424aaa2bc083efe3d0" gracePeriod=30 Nov 04 05:03:31.769807 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:31.769777 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg"] Nov 04 05:03:31.769955 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:31.769936 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" podUID="b9cd31cb-c46f-4e16-b289-d2546b1e6d9c" containerName="istio-proxy" containerID="cri-o://971faaa13fc0e1ddaeeee871578543ecce8edaf920044995631d7dbb13b4894a" gracePeriod=30 Nov 04 05:03:31.778536 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:31.778511 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q"] Nov 04 05:03:31.778684 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:31.778665 4395 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" podUID="e2f924d7-f797-4ed5-b57d-bb2271751fed" containerName="istio-proxy" containerID="cri-o://7b44bd1edc430eece9f1fab445d13ad78d2f633b65b5f692e438e206688c9819" gracePeriod=30 Nov 04 05:03:33.457362 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:03:33.457329 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 05:03:33.664586 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:33.664554 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-8bb5875fc-dct4m_744859c8-b8e3-4fc5-b31c-de2bcec377dc/authorino/0.log" Nov 04 05:03:36.823326 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:36.823283 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 04 05:03:36.823646 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:36.823350 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" podUID="e2f924d7-f797-4ed5-b57d-bb2271751fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 04 05:03:36.935397 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:36.934806 4395 patch_prober.go:29] interesting pod/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 04 05:03:36.935397 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:36.934880 4395 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" podUID="b9cd31cb-c46f-4e16-b289-d2546b1e6d9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 04 05:03:37.048074 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.048048 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" Nov 04 05:03:37.074675 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.074652 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" Nov 04 05:03:37.077310 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.077277 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" Nov 04 05:03:37.097561 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.097536 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2f924d7-f797-4ed5-b57d-bb2271751fed-kube-api-access-bpbvb\" (UniqueName: \"kubernetes.io/projected/e2f924d7-f797-4ed5-b57d-bb2271751fed-kube-api-access-bpbvb\") pod \"e2f924d7-f797-4ed5-b57d-bb2271751fed\" (UID: \"e2f924d7-f797-4ed5-b57d-bb2271751fed\") " Nov 04 05:03:37.097709 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.097596 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2f924d7-f797-4ed5-b57d-bb2271751fed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f924d7-f797-4ed5-b57d-bb2271751fed-workload-socket\") pod \"e2f924d7-f797-4ed5-b57d-bb2271751fed\" (UID: \"e2f924d7-f797-4ed5-b57d-bb2271751fed\") " Nov 04 05:03:37.097709 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.097650 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2f924d7-f797-4ed5-b57d-bb2271751fed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2f924d7-f797-4ed5-b57d-bb2271751fed-istio-podinfo\") pod \"e2f924d7-f797-4ed5-b57d-bb2271751fed\" (UID: \"e2f924d7-f797-4ed5-b57d-bb2271751fed\") " Nov 04 05:03:37.097709 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.097683 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2f924d7-f797-4ed5-b57d-bb2271751fed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2f924d7-f797-4ed5-b57d-bb2271751fed-istiod-ca-cert\") pod \"e2f924d7-f797-4ed5-b57d-bb2271751fed\" (UID: \"e2f924d7-f797-4ed5-b57d-bb2271751fed\") " Nov 04 05:03:37.097905 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.097716 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2f924d7-f797-4ed5-b57d-bb2271751fed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2f924d7-f797-4ed5-b57d-bb2271751fed-workload-certs\") pod \"e2f924d7-f797-4ed5-b57d-bb2271751fed\" (UID: \"e2f924d7-f797-4ed5-b57d-bb2271751fed\") " Nov 04 05:03:37.097905 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.097748 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2f924d7-f797-4ed5-b57d-bb2271751fed-istio-token\" (UniqueName: \"kubernetes.io/projected/e2f924d7-f797-4ed5-b57d-bb2271751fed-istio-token\") pod \"e2f924d7-f797-4ed5-b57d-bb2271751fed\" (UID: \"e2f924d7-f797-4ed5-b57d-bb2271751fed\") " Nov 04 05:03:37.097905 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.097786 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2f924d7-f797-4ed5-b57d-bb2271751fed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2f924d7-f797-4ed5-b57d-bb2271751fed-istio-envoy\") pod \"e2f924d7-f797-4ed5-b57d-bb2271751fed\" (UID: \"e2f924d7-f797-4ed5-b57d-bb2271751fed\") " Nov 04 05:03:37.097905 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.097849 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2f924d7-f797-4ed5-b57d-bb2271751fed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f924d7-f797-4ed5-b57d-bb2271751fed-credential-socket\") pod \"e2f924d7-f797-4ed5-b57d-bb2271751fed\" (UID: \"e2f924d7-f797-4ed5-b57d-bb2271751fed\") " Nov 04 05:03:37.098098 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.097938 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2f924d7-f797-4ed5-b57d-bb2271751fed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2f924d7-f797-4ed5-b57d-bb2271751fed-istio-data\") pod \"e2f924d7-f797-4ed5-b57d-bb2271751fed\" (UID: \"e2f924d7-f797-4ed5-b57d-bb2271751fed\") " Nov 04 05:03:37.098098 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.097989 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2f924d7-f797-4ed5-b57d-bb2271751fed-workload-socket" pod "e2f924d7-f797-4ed5-b57d-bb2271751fed" (UID: "e2f924d7-f797-4ed5-b57d-bb2271751fed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:03:37.098098 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.098067 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2f924d7-f797-4ed5-b57d-bb2271751fed-istiod-ca-cert" pod "e2f924d7-f797-4ed5-b57d-bb2271751fed" (UID: "e2f924d7-f797-4ed5-b57d-bb2271751fed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:03:37.098542 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.098272 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f924d7-f797-4ed5-b57d-bb2271751fed-workload-socket\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:37.098542 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.098282 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2f924d7-f797-4ed5-b57d-bb2271751fed-credential-socket" pod "e2f924d7-f797-4ed5-b57d-bb2271751fed" (UID: "e2f924d7-f797-4ed5-b57d-bb2271751fed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:03:37.098542 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.098294 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2f924d7-f797-4ed5-b57d-bb2271751fed-istiod-ca-cert\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:37.098542 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.098494 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2f924d7-f797-4ed5-b57d-bb2271751fed-istio-data" pod "e2f924d7-f797-4ed5-b57d-bb2271751fed" (UID: "e2f924d7-f797-4ed5-b57d-bb2271751fed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:03:37.098542 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.098504 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2f924d7-f797-4ed5-b57d-bb2271751fed-workload-certs" pod "e2f924d7-f797-4ed5-b57d-bb2271751fed" (UID: "e2f924d7-f797-4ed5-b57d-bb2271751fed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:03:37.100197 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.100171 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2f924d7-f797-4ed5-b57d-bb2271751fed-istio-token" pod "e2f924d7-f797-4ed5-b57d-bb2271751fed" (UID: "e2f924d7-f797-4ed5-b57d-bb2271751fed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:03:37.100378 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.100356 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2f924d7-f797-4ed5-b57d-bb2271751fed-istio-podinfo" pod "e2f924d7-f797-4ed5-b57d-bb2271751fed" (UID: "e2f924d7-f797-4ed5-b57d-bb2271751fed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:03:37.100483 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.100404 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2f924d7-f797-4ed5-b57d-bb2271751fed-kube-api-access-bpbvb" pod "e2f924d7-f797-4ed5-b57d-bb2271751fed" (UID: "e2f924d7-f797-4ed5-b57d-bb2271751fed"). InnerVolumeSpecName "kube-api-access-bpbvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:03:37.100534 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.100503 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2f924d7-f797-4ed5-b57d-bb2271751fed-istio-envoy" pod "e2f924d7-f797-4ed5-b57d-bb2271751fed" (UID: "e2f924d7-f797-4ed5-b57d-bb2271751fed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:03:37.101451 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.101428 4395 generic.go:356] "Generic (PLEG): container finished" podID="5798fb27-7552-42f7-ae48-c4ff7decff83" containerID="f689453b02d1e5b4f217640b7db32540b25105755d4f44424aaa2bc083efe3d0" exitCode=0 Nov 04 05:03:37.101561 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.101488 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" Nov 04 05:03:37.101561 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.101513 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" event={"ID":"5798fb27-7552-42f7-ae48-c4ff7decff83","Type":"ContainerDied","Data":"f689453b02d1e5b4f217640b7db32540b25105755d4f44424aaa2bc083efe3d0"} Nov 04 05:03:37.101675 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.101567 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj" event={"ID":"5798fb27-7552-42f7-ae48-c4ff7decff83","Type":"ContainerDied","Data":"7fec7cd4e3e7b21e6ce146a595f49a7786b65ef8aa19b0bde9a93c1caf36fdc4"} Nov 04 05:03:37.101675 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.101594 4395 scope.go:122] "RemoveContainer" containerID="f689453b02d1e5b4f217640b7db32540b25105755d4f44424aaa2bc083efe3d0" Nov 04 05:03:37.102836 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.102814 4395 generic.go:356] "Generic (PLEG): container finished" podID="e2f924d7-f797-4ed5-b57d-bb2271751fed" containerID="7b44bd1edc430eece9f1fab445d13ad78d2f633b65b5f692e438e206688c9819" exitCode=0 Nov 04 05:03:37.102936 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.102857 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" Nov 04 05:03:37.102936 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.102906 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" event={"ID":"e2f924d7-f797-4ed5-b57d-bb2271751fed","Type":"ContainerDied","Data":"7b44bd1edc430eece9f1fab445d13ad78d2f633b65b5f692e438e206688c9819"} Nov 04 05:03:37.103043 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.102953 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q" event={"ID":"e2f924d7-f797-4ed5-b57d-bb2271751fed","Type":"ContainerDied","Data":"e45b7d7f0c490bfeff9dcabd152a8efe1dfcd66accb064ab94963f7186cbf5a0"} Nov 04 05:03:37.104492 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.104470 4395 generic.go:356] "Generic (PLEG): container finished" podID="b9cd31cb-c46f-4e16-b289-d2546b1e6d9c" containerID="971faaa13fc0e1ddaeeee871578543ecce8edaf920044995631d7dbb13b4894a" exitCode=0 Nov 04 05:03:37.104602 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.104512 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" event={"ID":"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c","Type":"ContainerDied","Data":"971faaa13fc0e1ddaeeee871578543ecce8edaf920044995631d7dbb13b4894a"} Nov 04 05:03:37.104602 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.104524 4395 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" Nov 04 05:03:37.104602 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.104547 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg" event={"ID":"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c","Type":"ContainerDied","Data":"e09ecdc4ca97bbb09c19d4d3a2b76ac9ebd14ea21e801a94f8f4e8ff54f7c7b0"} Nov 04 05:03:37.110954 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.110935 4395 scope.go:122] "RemoveContainer" containerID="f9e1e716d3e3904299a6cf769a4559d76a778f432282c57bea1b14556a26994b" Nov 04 05:03:37.118630 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.118613 4395 scope.go:122] "RemoveContainer" containerID="f689453b02d1e5b4f217640b7db32540b25105755d4f44424aaa2bc083efe3d0" Nov 04 05:03:37.118875 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:03:37.118855 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f689453b02d1e5b4f217640b7db32540b25105755d4f44424aaa2bc083efe3d0\": container with ID starting with f689453b02d1e5b4f217640b7db32540b25105755d4f44424aaa2bc083efe3d0 not found: ID does not exist" containerID="f689453b02d1e5b4f217640b7db32540b25105755d4f44424aaa2bc083efe3d0" Nov 04 05:03:37.118937 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.118881 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f689453b02d1e5b4f217640b7db32540b25105755d4f44424aaa2bc083efe3d0"} err="failed to get container status \"f689453b02d1e5b4f217640b7db32540b25105755d4f44424aaa2bc083efe3d0\": rpc error: code = NotFound desc = could not find container \"f689453b02d1e5b4f217640b7db32540b25105755d4f44424aaa2bc083efe3d0\": container with ID starting with f689453b02d1e5b4f217640b7db32540b25105755d4f44424aaa2bc083efe3d0 not found: ID does not exist" Nov 04 05:03:37.118937 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.118896 4395 scope.go:122] "RemoveContainer" containerID="f9e1e716d3e3904299a6cf769a4559d76a778f432282c57bea1b14556a26994b" Nov 04 05:03:37.119135 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:03:37.119118 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9e1e716d3e3904299a6cf769a4559d76a778f432282c57bea1b14556a26994b\": container with ID starting with f9e1e716d3e3904299a6cf769a4559d76a778f432282c57bea1b14556a26994b not found: ID does not exist" containerID="f9e1e716d3e3904299a6cf769a4559d76a778f432282c57bea1b14556a26994b" Nov 04 05:03:37.119179 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.119139 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9e1e716d3e3904299a6cf769a4559d76a778f432282c57bea1b14556a26994b"} err="failed to get container status \"f9e1e716d3e3904299a6cf769a4559d76a778f432282c57bea1b14556a26994b\": rpc error: code = NotFound desc = could not find container \"f9e1e716d3e3904299a6cf769a4559d76a778f432282c57bea1b14556a26994b\": container with ID starting with f9e1e716d3e3904299a6cf769a4559d76a778f432282c57bea1b14556a26994b not found: ID does not exist" Nov 04 05:03:37.119179 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.119152 4395 scope.go:122] "RemoveContainer" containerID="7b44bd1edc430eece9f1fab445d13ad78d2f633b65b5f692e438e206688c9819" Nov 04 05:03:37.126608 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.126591 4395 scope.go:122] "RemoveContainer" containerID="7b44bd1edc430eece9f1fab445d13ad78d2f633b65b5f692e438e206688c9819" Nov 04 05:03:37.126843 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:03:37.126825 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b44bd1edc430eece9f1fab445d13ad78d2f633b65b5f692e438e206688c9819\": container with ID starting with 7b44bd1edc430eece9f1fab445d13ad78d2f633b65b5f692e438e206688c9819 not found: ID does not exist" containerID="7b44bd1edc430eece9f1fab445d13ad78d2f633b65b5f692e438e206688c9819" Nov 04 05:03:37.126906 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.126850 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b44bd1edc430eece9f1fab445d13ad78d2f633b65b5f692e438e206688c9819"} err="failed to get container status \"7b44bd1edc430eece9f1fab445d13ad78d2f633b65b5f692e438e206688c9819\": rpc error: code = NotFound desc = could not find container \"7b44bd1edc430eece9f1fab445d13ad78d2f633b65b5f692e438e206688c9819\": container with ID starting with 7b44bd1edc430eece9f1fab445d13ad78d2f633b65b5f692e438e206688c9819 not found: ID does not exist" Nov 04 05:03:37.126906 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.126864 4395 scope.go:122] "RemoveContainer" containerID="971faaa13fc0e1ddaeeee871578543ecce8edaf920044995631d7dbb13b4894a" Nov 04 05:03:37.134938 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.134923 4395 scope.go:122] "RemoveContainer" containerID="971faaa13fc0e1ddaeeee871578543ecce8edaf920044995631d7dbb13b4894a" Nov 04 05:03:37.135171 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:03:37.135155 4395 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"971faaa13fc0e1ddaeeee871578543ecce8edaf920044995631d7dbb13b4894a\": container with ID starting with 971faaa13fc0e1ddaeeee871578543ecce8edaf920044995631d7dbb13b4894a not found: ID does not exist" containerID="971faaa13fc0e1ddaeeee871578543ecce8edaf920044995631d7dbb13b4894a" Nov 04 05:03:37.135228 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.135178 4395 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"971faaa13fc0e1ddaeeee871578543ecce8edaf920044995631d7dbb13b4894a"} err="failed to get container status \"971faaa13fc0e1ddaeeee871578543ecce8edaf920044995631d7dbb13b4894a\": rpc error: code = NotFound desc = could not find container \"971faaa13fc0e1ddaeeee871578543ecce8edaf920044995631d7dbb13b4894a\": container with ID starting with 971faaa13fc0e1ddaeeee871578543ecce8edaf920044995631d7dbb13b4894a not found: ID does not exist" Nov 04 05:03:37.198887 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.198861 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5798fb27-7552-42f7-ae48-c4ff7decff83-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5798fb27-7552-42f7-ae48-c4ff7decff83-istio-data\") pod \"5798fb27-7552-42f7-ae48-c4ff7decff83\" (UID: \"5798fb27-7552-42f7-ae48-c4ff7decff83\") " Nov 04 05:03:37.199010 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.198895 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-kube-api-access-skgg5\" (UniqueName: \"kubernetes.io/projected/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-kube-api-access-skgg5\") pod \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\" (UID: \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\") " Nov 04 05:03:37.199010 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.198922 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5798fb27-7552-42f7-ae48-c4ff7decff83-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5798fb27-7552-42f7-ae48-c4ff7decff83-istio-envoy\") pod \"5798fb27-7552-42f7-ae48-c4ff7decff83\" (UID: \"5798fb27-7552-42f7-ae48-c4ff7decff83\") " Nov 04 05:03:37.199133 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.198942 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-istio-token\" (UniqueName: \"kubernetes.io/projected/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-istio-token\") pod \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\" (UID: \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\") " Nov 04 05:03:37.203961 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.199359 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5798fb27-7552-42f7-ae48-c4ff7decff83-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5798fb27-7552-42f7-ae48-c4ff7decff83-credential-socket\") pod \"5798fb27-7552-42f7-ae48-c4ff7decff83\" (UID: \"5798fb27-7552-42f7-ae48-c4ff7decff83\") " Nov 04 05:03:37.203961 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.199499 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-istio-data\") pod \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\" (UID: \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\") " Nov 04 05:03:37.203961 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.199528 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5798fb27-7552-42f7-ae48-c4ff7decff83-istio-data" pod "5798fb27-7552-42f7-ae48-c4ff7decff83" (UID: "5798fb27-7552-42f7-ae48-c4ff7decff83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:03:37.203961 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.199544 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5798fb27-7552-42f7-ae48-c4ff7decff83-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5798fb27-7552-42f7-ae48-c4ff7decff83-istiod-ca-cert\") pod \"5798fb27-7552-42f7-ae48-c4ff7decff83\" (UID: \"5798fb27-7552-42f7-ae48-c4ff7decff83\") " Nov 04 05:03:37.203961 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.199583 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-istiod-ca-cert\") pod \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\" (UID: \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\") " Nov 04 05:03:37.203961 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.199617 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-credential-socket\") pod \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\" (UID: \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\") " Nov 04 05:03:37.203961 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.199744 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5798fb27-7552-42f7-ae48-c4ff7decff83-credential-socket" pod "5798fb27-7552-42f7-ae48-c4ff7decff83" (UID: "5798fb27-7552-42f7-ae48-c4ff7decff83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:03:37.203961 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.200042 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5798fb27-7552-42f7-ae48-c4ff7decff83-istiod-ca-cert" pod "5798fb27-7552-42f7-ae48-c4ff7decff83" (UID: "5798fb27-7552-42f7-ae48-c4ff7decff83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:03:37.203961 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.200261 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-istio-data" pod "b9cd31cb-c46f-4e16-b289-d2546b1e6d9c" (UID: "b9cd31cb-c46f-4e16-b289-d2546b1e6d9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:03:37.203961 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.200555 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-istiod-ca-cert" pod "b9cd31cb-c46f-4e16-b289-d2546b1e6d9c" (UID: "b9cd31cb-c46f-4e16-b289-d2546b1e6d9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 05:03:37.203961 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.200768 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-credential-socket" pod "b9cd31cb-c46f-4e16-b289-d2546b1e6d9c" (UID: "b9cd31cb-c46f-4e16-b289-d2546b1e6d9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:03:37.203961 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.200869 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-istio-podinfo\") pod \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\" (UID: \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\") " Nov 04 05:03:37.203961 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.200917 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5798fb27-7552-42f7-ae48-c4ff7decff83-istio-token\" (UniqueName: \"kubernetes.io/projected/5798fb27-7552-42f7-ae48-c4ff7decff83-istio-token\") pod \"5798fb27-7552-42f7-ae48-c4ff7decff83\" (UID: \"5798fb27-7552-42f7-ae48-c4ff7decff83\") " Nov 04 05:03:37.203961 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.200971 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5798fb27-7552-42f7-ae48-c4ff7decff83-kube-api-access-tkjqb\" (UniqueName: \"kubernetes.io/projected/5798fb27-7552-42f7-ae48-c4ff7decff83-kube-api-access-tkjqb\") pod \"5798fb27-7552-42f7-ae48-c4ff7decff83\" (UID: \"5798fb27-7552-42f7-ae48-c4ff7decff83\") " Nov 04 05:03:37.203961 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.201015 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5798fb27-7552-42f7-ae48-c4ff7decff83-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5798fb27-7552-42f7-ae48-c4ff7decff83-workload-socket\") pod \"5798fb27-7552-42f7-ae48-c4ff7decff83\" (UID: \"5798fb27-7552-42f7-ae48-c4ff7decff83\") " Nov 04 05:03:37.203961 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.201055 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5798fb27-7552-42f7-ae48-c4ff7decff83-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5798fb27-7552-42f7-ae48-c4ff7decff83-istio-podinfo\") pod \"5798fb27-7552-42f7-ae48-c4ff7decff83\" (UID: \"5798fb27-7552-42f7-ae48-c4ff7decff83\") " Nov 04 05:03:37.203961 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.201110 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-workload-certs\") pod \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\" (UID: \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\") " Nov 04 05:03:37.203961 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.201146 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-workload-socket\") pod \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\" (UID: \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\") " Nov 04 05:03:37.203961 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.201190 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5798fb27-7552-42f7-ae48-c4ff7decff83-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5798fb27-7552-42f7-ae48-c4ff7decff83-workload-certs\") pod \"5798fb27-7552-42f7-ae48-c4ff7decff83\" (UID: \"5798fb27-7552-42f7-ae48-c4ff7decff83\") " Nov 04 05:03:37.203961 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.201247 4395 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-istio-envoy\") pod \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\" (UID: \"b9cd31cb-c46f-4e16-b289-d2546b1e6d9c\") " Nov 04 05:03:37.203961 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.201780 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-istio-token" pod "b9cd31cb-c46f-4e16-b289-d2546b1e6d9c" (UID: "b9cd31cb-c46f-4e16-b289-d2546b1e6d9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:03:37.203961 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.201936 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-istio-podinfo" pod "b9cd31cb-c46f-4e16-b289-d2546b1e6d9c" (UID: "b9cd31cb-c46f-4e16-b289-d2546b1e6d9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:03:37.208374 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.202753 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5798fb27-7552-42f7-ae48-c4ff7decff83-istio-envoy" pod "5798fb27-7552-42f7-ae48-c4ff7decff83" (UID: "5798fb27-7552-42f7-ae48-c4ff7decff83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:03:37.208374 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.202878 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-workload-socket" pod "b9cd31cb-c46f-4e16-b289-d2546b1e6d9c" (UID: "b9cd31cb-c46f-4e16-b289-d2546b1e6d9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:03:37.208374 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.202968 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5798fb27-7552-42f7-ae48-c4ff7decff83-workload-certs" pod "5798fb27-7552-42f7-ae48-c4ff7decff83" (UID: "5798fb27-7552-42f7-ae48-c4ff7decff83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:03:37.208374 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.203091 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-workload-certs" pod "b9cd31cb-c46f-4e16-b289-d2546b1e6d9c" (UID: "b9cd31cb-c46f-4e16-b289-d2546b1e6d9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:03:37.208374 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.204078 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-istio-envoy" pod "b9cd31cb-c46f-4e16-b289-d2546b1e6d9c" (UID: "b9cd31cb-c46f-4e16-b289-d2546b1e6d9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:03:37.208374 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.204331 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5798fb27-7552-42f7-ae48-c4ff7decff83-workload-socket" pod "5798fb27-7552-42f7-ae48-c4ff7decff83" (UID: "5798fb27-7552-42f7-ae48-c4ff7decff83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 05:03:37.208374 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.202650 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f924d7-f797-4ed5-b57d-bb2271751fed-credential-socket\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:37.208374 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.204471 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2f924d7-f797-4ed5-b57d-bb2271751fed-istio-data\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:37.208374 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.204489 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5798fb27-7552-42f7-ae48-c4ff7decff83-istio-data\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:37.208374 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.204506 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpbvb\" (UniqueName: \"kubernetes.io/projected/e2f924d7-f797-4ed5-b57d-bb2271751fed-kube-api-access-bpbvb\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:37.208374 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.204521 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-istio-token\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:37.208374 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.204541 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5798fb27-7552-42f7-ae48-c4ff7decff83-credential-socket\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:37.208374 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.204555 4395 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-istio-data\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:37.208374 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.204569 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5798fb27-7552-42f7-ae48-c4ff7decff83-istiod-ca-cert\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:37.208374 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.204585 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2f924d7-f797-4ed5-b57d-bb2271751fed-istio-podinfo\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:37.208374 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.204605 4395 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-istiod-ca-cert\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:37.208374 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.204620 4395 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-credential-socket\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:37.208374 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.204635 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2f924d7-f797-4ed5-b57d-bb2271751fed-workload-certs\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:37.208374 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.204649 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2f924d7-f797-4ed5-b57d-bb2271751fed-istio-token\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:37.208374 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.204667 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2f924d7-f797-4ed5-b57d-bb2271751fed-istio-envoy\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:37.208374 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.206947 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5798fb27-7552-42f7-ae48-c4ff7decff83-istio-podinfo" pod "5798fb27-7552-42f7-ae48-c4ff7decff83" (UID: "5798fb27-7552-42f7-ae48-c4ff7decff83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 05:03:37.208374 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.206952 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5798fb27-7552-42f7-ae48-c4ff7decff83-istio-token" pod "5798fb27-7552-42f7-ae48-c4ff7decff83" (UID: "5798fb27-7552-42f7-ae48-c4ff7decff83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:03:37.211515 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.211481 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5798fb27-7552-42f7-ae48-c4ff7decff83-kube-api-access-tkjqb" pod "5798fb27-7552-42f7-ae48-c4ff7decff83" (UID: "5798fb27-7552-42f7-ae48-c4ff7decff83"). InnerVolumeSpecName "kube-api-access-tkjqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:03:37.216327 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.212922 4395 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-kube-api-access-skgg5" pod "b9cd31cb-c46f-4e16-b289-d2546b1e6d9c" (UID: "b9cd31cb-c46f-4e16-b289-d2546b1e6d9c"). InnerVolumeSpecName "kube-api-access-skgg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 05:03:37.305551 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.305529 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-istio-podinfo\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:37.305551 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.305552 4395 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5798fb27-7552-42f7-ae48-c4ff7decff83-istio-token\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:37.305707 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.305563 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkjqb\" (UniqueName: \"kubernetes.io/projected/5798fb27-7552-42f7-ae48-c4ff7decff83-kube-api-access-tkjqb\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:37.305707 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.305573 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5798fb27-7552-42f7-ae48-c4ff7decff83-workload-socket\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:37.305707 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.305582 4395 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5798fb27-7552-42f7-ae48-c4ff7decff83-istio-podinfo\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:37.305707 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.305590 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-workload-certs\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:37.305707 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.305599 4395 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-workload-socket\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:37.305707 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.305607 4395 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5798fb27-7552-42f7-ae48-c4ff7decff83-workload-certs\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:37.305707 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.305616 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-istio-envoy\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:37.305707 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.305624 4395 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skgg5\" (UniqueName: \"kubernetes.io/projected/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c-kube-api-access-skgg5\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:37.305707 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.305632 4395 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5798fb27-7552-42f7-ae48-c4ff7decff83-istio-envoy\") on node \"ip-10-0-134-224.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 05:03:37.316520 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.316495 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q"] Nov 04 05:03:37.318781 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.318759 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-c6w5q"] Nov 04 05:03:37.646100 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.646027 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg"] Nov 04 05:03:37.649935 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.649910 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-2knzg"] Nov 04 05:03:37.751833 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.751799 4395 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj"] Nov 04 05:03:37.755785 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:37.755762 4395 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-a48f7e-openshift-default-b5476c79-68mxj"] Nov 04 05:03:38.415487 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:38.415458 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5798fb27-7552-42f7-ae48-c4ff7decff83" path="/var/lib/kubelet/pods/5798fb27-7552-42f7-ae48-c4ff7decff83/volumes" Nov 04 05:03:38.415923 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:38.415907 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9cd31cb-c46f-4e16-b289-d2546b1e6d9c" path="/var/lib/kubelet/pods/b9cd31cb-c46f-4e16-b289-d2546b1e6d9c/volumes" Nov 04 05:03:38.416336 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:38.416322 4395 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2f924d7-f797-4ed5-b57d-bb2271751fed" path="/var/lib/kubelet/pods/e2f924d7-f797-4ed5-b57d-bb2271751fed/volumes" Nov 04 05:03:47.041359 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:47.041329 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6c844db6b-c5mnw_126bfc55-f2af-4189-8ee0-3b383e4789e8/manager/0.log" Nov 04 05:03:47.641235 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:47.641205 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5gpsn_b6efbb1d-ceb4-4612-9ed2-266f9c8b43da/manager/0.log" Nov 04 05:03:49.334609 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:49.334580 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-09028ce6-ktmhx_8e40da9e-fed2-40ab-b430-9e69b8bab9d0/revoke-keys/0.log" Nov 04 05:03:49.562941 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:49.562915 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-11f35505-l69st_d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed/revoke-keys/0.log" Nov 04 05:03:49.777196 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:49.777133 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-157af145-tt25h_906b0195-e1ec-4930-89b1-3b4fb3c1bbdf/revoke-keys/0.log" Nov 04 05:03:50.452763 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:50.452733 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-5e8560ec-8l27l_4739a60d-77e7-4164-a229-ffd5122ce861/revoke-keys/0.log" Nov 04 05:03:51.124202 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:51.124176 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-919dcb68-cpk9z_029b0ad5-7427-432a-8f07-03c6646a300a/revoke-keys/0.log" Nov 04 05:03:51.569733 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:51.569702 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ad6d2d59-8hwjz_98988f22-9f65-416d-b315-95a330fc74ad/revoke-keys/0.log" Nov 04 05:03:52.208921 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:52.208897 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f8c59c1d-5mgkr_de211e54-1623-4eb4-b0b8-0c68323f373a/revoke-keys/0.log" Nov 04 05:03:54.371093 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:54.371065 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-8bb5875fc-dct4m_744859c8-b8e3-4fc5-b31c-de2bcec377dc/authorino/0.log" Nov 04 05:03:54.975258 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:54.975228 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-85ftp_8977878c-4c57-453c-9dab-428bd7d14140/manager/0.log" Nov 04 05:03:55.941495 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:55.941442 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-5g2bm_846fda69-90d5-463f-b306-40b443e8fbb7/manager/0.log" Nov 04 05:03:58.181414 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:58.181383 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-655d896fb8-qsdbq_dcc5b454-668a-4330-9f05-3e1c256bf81e/kube-auth-proxy/0.log" Nov 04 05:03:58.404421 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:03:58.404395 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-7bc9z_b948fdc0-b375-455d-8669-d47c9d84e8de/istio-proxy/0.log" Nov 04 05:04:00.564533 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:04:00.564506 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs_cfa8140b-9e2c-418e-bee1-6394bc7a6ac7/main/0.log" Nov 04 05:04:00.841354 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:04:00.841276 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-rzts7_102341e6-1ba0-40cd-b185-9f79f0310814/main/0.log" Nov 04 05:04:01.785726 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:04:01.785696 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-ks5ds_1fbe78fb-a1b4-4faa-8f86-e7df56474b9a/main/0.log" Nov 04 05:04:03.406495 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:04:03.406462 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-8bb5875fc-dct4m_744859c8-b8e3-4fc5-b31c-de2bcec377dc/authorino/0.log" Nov 04 05:04:03.456265 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:04:03.456235 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 05:04:13.913722 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:04:13.913690 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6c844db6b-c5mnw_126bfc55-f2af-4189-8ee0-3b383e4789e8/manager/0.log" Nov 04 05:04:14.425751 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:04:14.425721 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5gpsn_b6efbb1d-ceb4-4612-9ed2-266f9c8b43da/manager/0.log" Nov 04 05:04:15.666960 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:04:15.666933 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-09028ce6-ktmhx_8e40da9e-fed2-40ab-b430-9e69b8bab9d0/revoke-keys/0.log" Nov 04 05:04:15.894640 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:04:15.894614 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-11f35505-l69st_d132b5f2-aa5c-46b0-b38c-4cf4b0ab5eed/revoke-keys/0.log" Nov 04 05:04:16.116612 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:04:16.116584 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-157af145-tt25h_906b0195-e1ec-4930-89b1-3b4fb3c1bbdf/revoke-keys/0.log" Nov 04 05:04:16.772576 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:04:16.772547 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-5e8560ec-8l27l_4739a60d-77e7-4164-a229-ffd5122ce861/revoke-keys/0.log" Nov 04 05:04:17.445949 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:04:17.445919 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-919dcb68-cpk9z_029b0ad5-7427-432a-8f07-03c6646a300a/revoke-keys/0.log" Nov 04 05:04:17.923643 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:04:17.923616 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ad6d2d59-8hwjz_98988f22-9f65-416d-b315-95a330fc74ad/revoke-keys/0.log" Nov 04 05:04:18.599499 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:04:18.599468 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f8c59c1d-5mgkr_de211e54-1623-4eb4-b0b8-0c68323f373a/revoke-keys/0.log" Nov 04 05:04:19.970717 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:04:19.970693 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-8bb5875fc-dct4m_744859c8-b8e3-4fc5-b31c-de2bcec377dc/authorino/0.log" Nov 04 05:04:20.599986 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:04:20.599955 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-85ftp_8977878c-4c57-453c-9dab-428bd7d14140/manager/0.log" Nov 04 05:04:21.071680 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:04:21.071652 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-5g2bm_846fda69-90d5-463f-b306-40b443e8fbb7/manager/0.log" Nov 04 05:04:23.140606 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:04:23.140575 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-655d896fb8-qsdbq_dcc5b454-668a-4330-9f05-3e1c256bf81e/kube-auth-proxy/0.log" Nov 04 05:04:23.376311 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:04:23.376272 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-7bc9z_b948fdc0-b375-455d-8669-d47c9d84e8de/istio-proxy/0.log" Nov 04 05:04:25.166725 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:04:25.166691 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-d79cs_cfa8140b-9e2c-418e-bee1-6394bc7a6ac7/main/0.log" Nov 04 05:04:25.442417 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:04:25.442345 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-rzts7_102341e6-1ba0-40cd-b185-9f79f0310814/main/0.log" Nov 04 05:04:26.382532 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:04:26.382501 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-ks5ds_1fbe78fb-a1b4-4faa-8f86-e7df56474b9a/main/0.log" Nov 04 05:04:33.457421 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:04:33.457390 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 05:04:49.643395 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:04:49.643315 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6c844db6b-c5mnw_126bfc55-f2af-4189-8ee0-3b383e4789e8/manager/0.log" Nov 04 05:04:50.188580 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:04:50.188543 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6c844db6b-c5mnw_126bfc55-f2af-4189-8ee0-3b383e4789e8/manager/0.log" Nov 04 05:05:03.457581 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:05:03.457546 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 05:05:23.400613 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:23.400576 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6c844db6b-c5mnw_126bfc55-f2af-4189-8ee0-3b383e4789e8/manager/0.log" Nov 04 05:05:23.938192 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:23.938163 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6c844db6b-c5mnw_126bfc55-f2af-4189-8ee0-3b383e4789e8/manager/0.log" Nov 04 05:05:33.456594 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:05:33.456555 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 05:05:35.000497 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:35.000452 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mwgm5_3e3bf36f-7156-4151-9d6d-0937169dbcca/global-pull-secret-syncer/0.log" Nov 04 05:05:38.031566 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:38.031539 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ct2mt_a7936867-0a26-4c2e-b94d-a1756cd14873/konnectivity-agent/0.log" Nov 04 05:05:39.283162 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:39.283123 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-224.us-west-2.compute.internal_88d814719e03f487d48b25be7946048f/haproxy/0.log" Nov 04 05:05:44.056957 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:44.056924 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-8bb5875fc-dct4m_744859c8-b8e3-4fc5-b31c-de2bcec377dc/authorino/0.log" Nov 04 05:05:44.388353 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:44.388260 4395 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-f7wrj/perf-node-gather-daemonset-phk4m"] Nov 04 05:05:44.388648 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:44.388634 4395 cpu_manager.go:414] "containerMap: removing container" podUID="b9cd31cb-c46f-4e16-b289-d2546b1e6d9c" containerName="istio-proxy" Nov 04 05:05:44.388690 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:44.388649 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9cd31cb-c46f-4e16-b289-d2546b1e6d9c" containerName="istio-proxy" Nov 04 05:05:44.388690 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:44.388663 4395 cpu_manager.go:414] "containerMap: removing container" podUID="e2f924d7-f797-4ed5-b57d-bb2271751fed" containerName="istio-proxy" Nov 04 05:05:44.388690 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:44.388669 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2f924d7-f797-4ed5-b57d-bb2271751fed" containerName="istio-proxy" Nov 04 05:05:44.388690 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:44.388676 4395 cpu_manager.go:414] "containerMap: removing container" podUID="5798fb27-7552-42f7-ae48-c4ff7decff83" containerName="istio-proxy" Nov 04 05:05:44.388690 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:44.388681 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5798fb27-7552-42f7-ae48-c4ff7decff83" containerName="istio-proxy" Nov 04 05:05:44.388840 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:44.388692 4395 cpu_manager.go:414] "containerMap: removing container" podUID="5798fb27-7552-42f7-ae48-c4ff7decff83" containerName="istio-proxy" Nov 04 05:05:44.388840 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:44.388697 4395 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5798fb27-7552-42f7-ae48-c4ff7decff83" containerName="istio-proxy" Nov 04 05:05:44.388840 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:44.388755 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9cd31cb-c46f-4e16-b289-d2546b1e6d9c" containerName="istio-proxy" Nov 04 05:05:44.388840 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:44.388763 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2f924d7-f797-4ed5-b57d-bb2271751fed" containerName="istio-proxy" Nov 04 05:05:44.388840 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:44.388770 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="5798fb27-7552-42f7-ae48-c4ff7decff83" containerName="istio-proxy" Nov 04 05:05:44.388840 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:44.388776 4395 memory_manager.go:365] "RemoveStaleState removing state" podUID="5798fb27-7552-42f7-ae48-c4ff7decff83" containerName="istio-proxy" Nov 04 05:05:44.391894 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:44.391874 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f7wrj/perf-node-gather-daemonset-phk4m" Nov 04 05:05:44.440892 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:44.440867 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f7wrj\"/\"openshift-service-ca.crt\"" Nov 04 05:05:44.475006 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:44.474986 4395 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f7wrj\"/\"kube-root-ca.crt\"" Nov 04 05:05:44.475108 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:44.475039 4395 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-f7wrj\"/\"default-dockercfg-kt2w6\"" Nov 04 05:05:44.488998 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:44.488975 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f7wrj/perf-node-gather-daemonset-phk4m"] Nov 04 05:05:44.500403 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:44.500379 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/32605eb2-fb67-43e4-8d2b-bbf993dcb6c0-proc\") pod \"perf-node-gather-daemonset-phk4m\" (UID: \"32605eb2-fb67-43e4-8d2b-bbf993dcb6c0\") " pod="openshift-must-gather-f7wrj/perf-node-gather-daemonset-phk4m" Nov 04 05:05:44.500501 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:44.500430 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32605eb2-fb67-43e4-8d2b-bbf993dcb6c0-sys\") pod \"perf-node-gather-daemonset-phk4m\" (UID: \"32605eb2-fb67-43e4-8d2b-bbf993dcb6c0\") " pod="openshift-must-gather-f7wrj/perf-node-gather-daemonset-phk4m" Nov 04 05:05:44.500571 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:44.500556 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/32605eb2-fb67-43e4-8d2b-bbf993dcb6c0-podres\") pod \"perf-node-gather-daemonset-phk4m\" (UID: \"32605eb2-fb67-43e4-8d2b-bbf993dcb6c0\") " pod="openshift-must-gather-f7wrj/perf-node-gather-daemonset-phk4m" Nov 04 05:05:44.500622 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:44.500604 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rtw5\" (UniqueName: \"kubernetes.io/projected/32605eb2-fb67-43e4-8d2b-bbf993dcb6c0-kube-api-access-6rtw5\") pod \"perf-node-gather-daemonset-phk4m\" (UID: \"32605eb2-fb67-43e4-8d2b-bbf993dcb6c0\") " pod="openshift-must-gather-f7wrj/perf-node-gather-daemonset-phk4m" Nov 04 05:05:44.500737 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:44.500716 4395 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32605eb2-fb67-43e4-8d2b-bbf993dcb6c0-lib-modules\") pod \"perf-node-gather-daemonset-phk4m\" (UID: \"32605eb2-fb67-43e4-8d2b-bbf993dcb6c0\") " pod="openshift-must-gather-f7wrj/perf-node-gather-daemonset-phk4m" Nov 04 05:05:44.601219 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:44.601195 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/32605eb2-fb67-43e4-8d2b-bbf993dcb6c0-proc\") pod \"perf-node-gather-daemonset-phk4m\" (UID: \"32605eb2-fb67-43e4-8d2b-bbf993dcb6c0\") " pod="openshift-must-gather-f7wrj/perf-node-gather-daemonset-phk4m" Nov 04 05:05:44.601350 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:44.601231 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32605eb2-fb67-43e4-8d2b-bbf993dcb6c0-sys\") pod \"perf-node-gather-daemonset-phk4m\" (UID: \"32605eb2-fb67-43e4-8d2b-bbf993dcb6c0\") " pod="openshift-must-gather-f7wrj/perf-node-gather-daemonset-phk4m" Nov 04 05:05:44.601350 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:44.601249 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/32605eb2-fb67-43e4-8d2b-bbf993dcb6c0-podres\") pod \"perf-node-gather-daemonset-phk4m\" (UID: \"32605eb2-fb67-43e4-8d2b-bbf993dcb6c0\") " pod="openshift-must-gather-f7wrj/perf-node-gather-daemonset-phk4m" Nov 04 05:05:44.601350 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:44.601284 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rtw5\" (UniqueName: \"kubernetes.io/projected/32605eb2-fb67-43e4-8d2b-bbf993dcb6c0-kube-api-access-6rtw5\") pod \"perf-node-gather-daemonset-phk4m\" (UID: \"32605eb2-fb67-43e4-8d2b-bbf993dcb6c0\") " pod="openshift-must-gather-f7wrj/perf-node-gather-daemonset-phk4m" Nov 04 05:05:44.601350 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:44.601329 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/32605eb2-fb67-43e4-8d2b-bbf993dcb6c0-proc\") pod \"perf-node-gather-daemonset-phk4m\" (UID: \"32605eb2-fb67-43e4-8d2b-bbf993dcb6c0\") " pod="openshift-must-gather-f7wrj/perf-node-gather-daemonset-phk4m" Nov 04 05:05:44.601350 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:44.601332 4395 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32605eb2-fb67-43e4-8d2b-bbf993dcb6c0-lib-modules\") pod \"perf-node-gather-daemonset-phk4m\" (UID: \"32605eb2-fb67-43e4-8d2b-bbf993dcb6c0\") " pod="openshift-must-gather-f7wrj/perf-node-gather-daemonset-phk4m" Nov 04 05:05:44.601350 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:44.601340 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32605eb2-fb67-43e4-8d2b-bbf993dcb6c0-sys\") pod \"perf-node-gather-daemonset-phk4m\" (UID: \"32605eb2-fb67-43e4-8d2b-bbf993dcb6c0\") " pod="openshift-must-gather-f7wrj/perf-node-gather-daemonset-phk4m" Nov 04 05:05:44.601585 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:44.601370 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/32605eb2-fb67-43e4-8d2b-bbf993dcb6c0-podres\") pod \"perf-node-gather-daemonset-phk4m\" (UID: \"32605eb2-fb67-43e4-8d2b-bbf993dcb6c0\") " pod="openshift-must-gather-f7wrj/perf-node-gather-daemonset-phk4m" Nov 04 05:05:44.601585 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:44.601382 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32605eb2-fb67-43e4-8d2b-bbf993dcb6c0-lib-modules\") pod \"perf-node-gather-daemonset-phk4m\" (UID: \"32605eb2-fb67-43e4-8d2b-bbf993dcb6c0\") " pod="openshift-must-gather-f7wrj/perf-node-gather-daemonset-phk4m" Nov 04 05:05:44.658532 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:44.658481 4395 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rtw5\" (UniqueName: \"kubernetes.io/projected/32605eb2-fb67-43e4-8d2b-bbf993dcb6c0-kube-api-access-6rtw5\") pod \"perf-node-gather-daemonset-phk4m\" (UID: \"32605eb2-fb67-43e4-8d2b-bbf993dcb6c0\") " pod="openshift-must-gather-f7wrj/perf-node-gather-daemonset-phk4m" Nov 04 05:05:44.705044 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:44.705026 4395 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f7wrj/perf-node-gather-daemonset-phk4m" Nov 04 05:05:44.804659 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:44.804632 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-85ftp_8977878c-4c57-453c-9dab-428bd7d14140/manager/0.log" Nov 04 05:05:45.077012 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:45.076991 4395 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f7wrj/perf-node-gather-daemonset-phk4m"] Nov 04 05:05:45.078742 ip-10-0-134-224 kubenswrapper[4395]: W1104 05:05:45.078713 4395 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod32605eb2_fb67_43e4_8d2b_bbf993dcb6c0.slice/crio-c71b790151cfd5ae565df159b1ae69a0a0f8922866d9a529469fd1f4d0b8ca42 WatchSource:0}: Error finding container c71b790151cfd5ae565df159b1ae69a0a0f8922866d9a529469fd1f4d0b8ca42: Status 404 returned error can't find the container with id c71b790151cfd5ae565df159b1ae69a0a0f8922866d9a529469fd1f4d0b8ca42 Nov 04 05:05:45.798622 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:45.798584 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f7wrj/perf-node-gather-daemonset-phk4m" event={"ID":"32605eb2-fb67-43e4-8d2b-bbf993dcb6c0","Type":"ContainerStarted","Data":"c4814efece39d969a358d2703900a7ec4a76f35d8578faac1138add10ed35cd4"} Nov 04 05:05:45.798787 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:45.798682 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-f7wrj/perf-node-gather-daemonset-phk4m" Nov 04 05:05:45.798787 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:45.798707 4395 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f7wrj/perf-node-gather-daemonset-phk4m" event={"ID":"32605eb2-fb67-43e4-8d2b-bbf993dcb6c0","Type":"ContainerStarted","Data":"c71b790151cfd5ae565df159b1ae69a0a0f8922866d9a529469fd1f4d0b8ca42"} Nov 04 05:05:45.888190 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:45.888160 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-5g2bm_846fda69-90d5-463f-b306-40b443e8fbb7/manager/0.log" Nov 04 05:05:45.908769 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:45.908712 4395 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-f7wrj/perf-node-gather-daemonset-phk4m" podStartSLOduration=1.908696908 podStartE2EDuration="1.908696908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 05:05:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 05:05:45.906521071 +0000 UTC m=+1187.596603519" watchObservedRunningTime="2026-11-04 05:05:45.908696908 +0000 UTC m=+1187.598779356" Nov 04 05:05:51.814712 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:51.814685 4395 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-f7wrj/perf-node-gather-daemonset-phk4m" Nov 04 05:05:54.183660 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:54.183632 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-m4bwb_343dcbb2-ac23-4978-a763-03964fd95eaf/monitoring-plugin/0.log" Nov 04 05:05:57.101249 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:57.101220 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wrqmt_b77f3812-c82c-4e4f-8cf3-571265f1629a/node-exporter/0.log" Nov 04 05:05:57.516356 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:57.516254 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wrqmt_b77f3812-c82c-4e4f-8cf3-571265f1629a/kube-rbac-proxy/0.log" Nov 04 05:05:57.936130 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:57.936107 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wrqmt_b77f3812-c82c-4e4f-8cf3-571265f1629a/init-textfile/0.log" Nov 04 05:05:59.670945 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:05:59.670914 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9a676d75-8ddf-46b5-8ee1-acdb7f41834d/prometheus/0.log" Nov 04 05:06:00.095103 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:06:00.095076 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9a676d75-8ddf-46b5-8ee1-acdb7f41834d/config-reloader/0.log" Nov 04 05:06:00.511499 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:06:00.511435 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9a676d75-8ddf-46b5-8ee1-acdb7f41834d/thanos-sidecar/0.log" Nov 04 05:06:00.928410 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:06:00.928385 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9a676d75-8ddf-46b5-8ee1-acdb7f41834d/kube-rbac-proxy-web/0.log" Nov 04 05:06:01.344118 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:06:01.344071 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9a676d75-8ddf-46b5-8ee1-acdb7f41834d/kube-rbac-proxy/0.log" Nov 04 05:06:01.759973 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:06:01.759911 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9a676d75-8ddf-46b5-8ee1-acdb7f41834d/kube-rbac-proxy-thanos/0.log" Nov 04 05:06:02.176536 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:06:02.176511 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9a676d75-8ddf-46b5-8ee1-acdb7f41834d/init-config-reloader/0.log" Nov 04 05:06:02.597360 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:06:02.597335 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-vnlrf_e0197f53-2bf1-4a3c-ae89-d8a74ff7c33c/prometheus-operator/0.log" Nov 04 05:06:03.010665 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:06:03.010601 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-vnlrf_e0197f53-2bf1-4a3c-ae89-d8a74ff7c33c/kube-rbac-proxy/0.log" Nov 04 05:06:03.456908 ip-10-0-134-224 kubenswrapper[4395]: E1104 05:06:03.456871 4395 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qn6tv" containerName="extractor" Nov 04 05:06:05.124101 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:06:05.124062 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75dcc4968f-ql7lm_369386da-4587-45d0-b344-0809b01f162b/thanos-query/0.log" Nov 04 05:06:05.541096 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:06:05.541015 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75dcc4968f-ql7lm_369386da-4587-45d0-b344-0809b01f162b/kube-rbac-proxy-web/0.log" Nov 04 05:06:05.959503 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:06:05.959451 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75dcc4968f-ql7lm_369386da-4587-45d0-b344-0809b01f162b/kube-rbac-proxy/0.log" Nov 04 05:06:06.380047 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:06:06.380016 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75dcc4968f-ql7lm_369386da-4587-45d0-b344-0809b01f162b/prom-label-proxy/0.log" Nov 04 05:06:06.794566 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:06:06.794543 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75dcc4968f-ql7lm_369386da-4587-45d0-b344-0809b01f162b/kube-rbac-proxy-rules/0.log" Nov 04 05:06:07.210401 ip-10-0-134-224 kubenswrapper[4395]: I1104 05:06:07.210325 4395 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75dcc4968f-ql7lm_369386da-4587-45d0-b344-0809b01f162b/kube-rbac-proxy-metrics/0.log"