Nov 04 03:16:41.110241 ip-10-0-133-6 systemd[1]: Starting Kubernetes Kubelet... Nov 04 03:16:41.521903 ip-10-0-133-6 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 03:16:41.521903 ip-10-0-133-6 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 04 03:16:41.521903 ip-10-0-133-6 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 03:16:41.521903 ip-10-0-133-6 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 03:16:41.528347 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528250 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:16:41.528347 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528346 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:16:41.528347 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528350 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:16:41.528347 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528353 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:16:41.528468 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528358 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:16:41.528468 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528360 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:16:41.528468 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528364 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:16:41.528468 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528367 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:16:41.528468 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528370 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:16:41.528468 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528372 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:16:41.528468 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528375 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:16:41.528468 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528378 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:16:41.528468 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528380 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:16:41.528468 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528383 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:16:41.528468 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528385 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:16:41.528468 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528388 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:16:41.528468 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528391 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:16:41.528468 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528393 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:16:41.528468 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528396 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:16:41.528468 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528398 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:16:41.528468 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528401 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:16:41.528468 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528407 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:16:41.528468 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528409 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:16:41.528468 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528412 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:16:41.528956 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528415 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:16:41.528956 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528417 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:16:41.528956 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528420 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:16:41.528956 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528423 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:16:41.528956 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528426 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:16:41.528956 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528429 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:16:41.528956 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528431 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:16:41.528956 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528434 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:16:41.528956 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528437 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:16:41.528956 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528439 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:16:41.528956 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528442 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:16:41.528956 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528445 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:16:41.528956 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528447 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:16:41.528956 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528451 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:16:41.528956 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528454 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:16:41.528956 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528456 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:16:41.528956 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528459 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:16:41.528956 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528462 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:16:41.528956 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528465 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:16:41.529415 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528467 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:16:41.529415 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528470 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:16:41.529415 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528473 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:16:41.529415 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528475 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:16:41.529415 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528478 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:16:41.529415 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528481 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:16:41.529415 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528484 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:16:41.529415 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528486 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:16:41.529415 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528489 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:16:41.529415 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528491 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:16:41.529415 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528494 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:16:41.529415 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528497 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:16:41.529415 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528499 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:16:41.529415 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528502 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:16:41.529415 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528505 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:16:41.529415 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528507 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:16:41.529415 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528510 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:16:41.529415 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528512 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:16:41.529415 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528514 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:16:41.529415 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528517 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:16:41.529879 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528520 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:16:41.529879 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528523 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:16:41.529879 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528525 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:16:41.529879 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528528 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:16:41.529879 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528530 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:16:41.529879 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528533 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:16:41.529879 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528538 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:16:41.529879 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528542 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:16:41.529879 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528546 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:16:41.529879 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528549 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:16:41.529879 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528552 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:16:41.529879 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528556 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:16:41.529879 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528560 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:16:41.529879 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528563 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:16:41.529879 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528566 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:16:41.529879 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528569 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:16:41.529879 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528572 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:16:41.529879 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528574 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:16:41.529879 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528577 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:16:41.530348 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528580 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:16:41.530348 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528583 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:16:41.530348 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528585 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:16:41.530348 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528588 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:16:41.530348 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528590 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:16:41.530348 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528593 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:16:41.530348 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528596 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:16:41.530348 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528598 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:16:41.530348 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528601 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:16:41.530348 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528604 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:16:41.530348 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528606 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:16:41.530348 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528609 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:16:41.530348 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528612 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:16:41.530348 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528615 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:16:41.530348 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528617 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:16:41.530348 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528620 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:16:41.530348 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528622 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:16:41.530348 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528625 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:16:41.530348 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528628 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:16:41.530348 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528631 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:16:41.530815 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528633 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:16:41.530815 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528636 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:16:41.530815 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528639 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:16:41.530815 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.528642 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:16:41.530815 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530446 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:16:41.530815 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530456 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:16:41.530815 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530460 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:16:41.530815 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530463 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:16:41.530815 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530466 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:16:41.530815 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530470 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:16:41.530815 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530472 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:16:41.530815 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530475 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:16:41.530815 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530477 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:16:41.530815 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530480 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:16:41.530815 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530483 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:16:41.530815 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530485 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:16:41.530815 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530488 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:16:41.530815 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530490 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:16:41.530815 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530493 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:16:41.530815 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530495 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:16:41.531304 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530498 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:16:41.531304 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530501 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:16:41.531304 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530503 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:16:41.531304 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530506 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:16:41.531304 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530508 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:16:41.531304 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530511 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:16:41.531304 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530514 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:16:41.531304 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530516 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:16:41.531304 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530519 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:16:41.531304 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530522 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:16:41.531304 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530524 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:16:41.531304 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530527 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:16:41.531304 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530529 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:16:41.531304 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530533 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:16:41.531304 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530535 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:16:41.531304 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530538 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:16:41.531304 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530541 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:16:41.531304 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530544 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:16:41.531304 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530548 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:16:41.531763 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530552 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:16:41.531763 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530555 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:16:41.531763 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530558 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:16:41.531763 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530562 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:16:41.531763 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530565 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:16:41.531763 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530567 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:16:41.531763 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530570 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:16:41.531763 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530573 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:16:41.531763 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530576 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:16:41.531763 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530578 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:16:41.531763 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530581 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:16:41.531763 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530583 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:16:41.531763 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530586 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:16:41.531763 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530589 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:16:41.531763 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530591 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:16:41.531763 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530594 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:16:41.531763 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530596 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:16:41.531763 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530599 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:16:41.531763 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530602 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:16:41.532232 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530605 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:16:41.532232 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530607 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:16:41.532232 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530610 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:16:41.532232 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530612 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:16:41.532232 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530615 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:16:41.532232 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530617 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:16:41.532232 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530620 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:16:41.532232 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530623 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:16:41.532232 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530626 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:16:41.532232 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530628 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:16:41.532232 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530631 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:16:41.532232 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530648 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:16:41.532232 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530660 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:16:41.532232 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530663 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:16:41.532232 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530666 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:16:41.532232 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530669 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:16:41.532232 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530672 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:16:41.532232 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530675 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:16:41.532232 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530677 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:16:41.532232 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530680 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:16:41.532700 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530683 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:16:41.532700 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530685 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:16:41.532700 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530688 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:16:41.532700 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530690 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:16:41.532700 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530693 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:16:41.532700 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530695 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:16:41.532700 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530698 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:16:41.532700 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530700 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:16:41.532700 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530703 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:16:41.532700 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530706 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:16:41.532700 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530708 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:16:41.532700 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530714 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:16:41.532700 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530717 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:16:41.532700 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530720 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:16:41.532700 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530722 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:16:41.532700 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530725 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:16:41.532700 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530727 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:16:41.532700 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530730 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:16:41.532700 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530732 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:16:41.533195 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530737 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:16:41.533195 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530741 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:16:41.533195 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530744 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:16:41.533195 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530746 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:16:41.533195 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530749 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:16:41.533195 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530753 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:16:41.533195 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530756 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:16:41.533195 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530758 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:16:41.533195 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530761 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:16:41.533195 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530764 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:16:41.533195 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530766 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:16:41.533195 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530770 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:16:41.533195 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.530772 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:16:41.533195 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531351 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:16:41.533195 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531358 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:16:41.533195 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531361 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:16:41.533195 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531365 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:16:41.533195 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531368 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:16:41.533195 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531370 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:16:41.533195 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531373 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:16:41.533657 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531375 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:16:41.533657 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531378 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:16:41.533657 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531382 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:16:41.533657 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531384 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:16:41.533657 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531388 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:16:41.533657 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531390 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:16:41.533657 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531393 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:16:41.533657 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531396 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:16:41.533657 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531398 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:16:41.533657 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531401 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:16:41.533657 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531403 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:16:41.533657 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531406 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:16:41.533657 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531409 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:16:41.533657 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531411 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:16:41.533657 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531414 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:16:41.533657 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531417 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:16:41.533657 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531419 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:16:41.533657 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531422 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:16:41.533657 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531425 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:16:41.533657 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531427 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:16:41.534124 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531430 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:16:41.534124 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531432 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:16:41.534124 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531435 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:16:41.534124 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531438 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:16:41.534124 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531440 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:16:41.534124 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531443 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:16:41.534124 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531445 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:16:41.534124 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531448 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:16:41.534124 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531451 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:16:41.534124 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531454 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:16:41.534124 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531456 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:16:41.534124 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531459 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:16:41.534124 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531462 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:16:41.534124 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531464 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:16:41.534124 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531467 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:16:41.534124 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531469 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:16:41.534124 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531472 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:16:41.534124 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531474 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:16:41.534124 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531477 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:16:41.534124 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531479 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:16:41.534627 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531481 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:16:41.534627 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531484 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:16:41.534627 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531486 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:16:41.534627 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531489 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:16:41.534627 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531493 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:16:41.534627 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531496 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:16:41.534627 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531499 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:16:41.534627 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531502 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:16:41.534627 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531504 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:16:41.534627 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531507 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:16:41.534627 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531510 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:16:41.534627 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531513 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:16:41.534627 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531515 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:16:41.534627 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531518 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:16:41.534627 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531521 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:16:41.534627 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531523 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:16:41.534627 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531525 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:16:41.534627 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531528 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:16:41.534627 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531530 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:16:41.535100 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531533 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:16:41.535100 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531537 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:16:41.535100 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531540 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:16:41.535100 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531542 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:16:41.535100 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531545 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:16:41.535100 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531548 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:16:41.535100 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531550 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:16:41.535100 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531553 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:16:41.535100 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531555 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:16:41.535100 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531558 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:16:41.535100 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531561 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:16:41.535100 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531563 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:16:41.535100 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531566 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:16:41.535100 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531568 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:16:41.535100 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531571 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:16:41.535100 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531573 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:16:41.535100 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531576 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:16:41.535100 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531578 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:16:41.535100 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531581 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:16:41.535537 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531585 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:16:41.535537 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531588 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:16:41.535537 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531592 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:16:41.535537 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531595 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:16:41.535537 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531598 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:16:41.535537 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531602 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:16:41.535537 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531605 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:16:41.535537 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531607 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:16:41.535537 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531610 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:16:41.535537 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531612 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:16:41.535537 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531615 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:16:41.535537 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531618 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:16:41.535537 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531621 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:16:41.535537 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531623 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:16:41.535537 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531627 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:16:41.535537 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531630 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:16:41.535537 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531634 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:16:41.535537 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531636 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:16:41.535537 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531639 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:16:41.536023 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531641 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:16:41.536023 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.531644 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:16:41.536023 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532637 2578 flags.go:64] FLAG: --address="0.0.0.0" Nov 04 03:16:41.536023 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532650 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 04 03:16:41.536023 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532657 2578 flags.go:64] FLAG: --anonymous-auth="true" Nov 04 03:16:41.536023 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532663 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 04 03:16:41.536023 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532667 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 04 03:16:41.536023 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532670 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 04 03:16:41.536023 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532675 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 04 03:16:41.536023 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532680 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 04 03:16:41.536023 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532683 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 04 03:16:41.536023 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532686 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 04 03:16:41.536023 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532689 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 04 03:16:41.536023 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532693 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 04 03:16:41.536023 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532696 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 04 03:16:41.536023 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532699 2578 flags.go:64] FLAG: --cgroup-root="" Nov 04 03:16:41.536023 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532703 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 04 03:16:41.536023 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532706 2578 flags.go:64] FLAG: --client-ca-file="" Nov 04 03:16:41.536023 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532709 2578 flags.go:64] FLAG: --cloud-provider="external" Nov 04 03:16:41.536023 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532720 2578 flags.go:64] FLAG: --cluster-dns="[]" Nov 04 03:16:41.536023 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532727 2578 flags.go:64] FLAG: --cluster-domain="" Nov 04 03:16:41.536023 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532730 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 04 03:16:41.536023 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532734 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 04 03:16:41.536546 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532737 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 04 03:16:41.536546 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532740 2578 flags.go:64] FLAG: --container-log-max-files="5" Nov 04 03:16:41.536546 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532745 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 04 03:16:41.536546 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532750 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 04 03:16:41.536546 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532754 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 04 03:16:41.536546 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532757 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 04 03:16:41.536546 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532760 2578 flags.go:64] FLAG: --contention-profiling="false" Nov 04 03:16:41.536546 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532763 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 04 03:16:41.536546 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532766 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 04 03:16:41.536546 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532769 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 04 03:16:41.536546 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532772 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 04 03:16:41.536546 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532777 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 04 03:16:41.536546 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532780 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 04 03:16:41.536546 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532782 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 04 03:16:41.536546 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532786 2578 flags.go:64] FLAG: --enable-load-reader="false" Nov 04 03:16:41.536546 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532789 2578 flags.go:64] FLAG: --enable-server="true" Nov 04 03:16:41.536546 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532792 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 04 03:16:41.536546 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532799 2578 flags.go:64] FLAG: --event-burst="100" Nov 04 03:16:41.536546 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532802 2578 flags.go:64] FLAG: --event-qps="50" Nov 04 03:16:41.536546 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532805 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 04 03:16:41.536546 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532808 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 04 03:16:41.536546 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532811 2578 flags.go:64] FLAG: --eviction-hard="" Nov 04 03:16:41.536546 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532815 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 04 03:16:41.536546 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532818 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 04 03:16:41.537196 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532821 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 04 03:16:41.537196 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532824 2578 flags.go:64] FLAG: --eviction-soft="" Nov 04 03:16:41.537196 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532827 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 04 03:16:41.537196 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532830 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 04 03:16:41.537196 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532833 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 04 03:16:41.537196 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532840 2578 flags.go:64] FLAG: --experimental-mounter-path="" Nov 04 03:16:41.537196 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532843 2578 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 04 03:16:41.537196 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532846 2578 flags.go:64] FLAG: --fail-swap-on="true" Nov 04 03:16:41.537196 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532849 2578 flags.go:64] FLAG: --feature-gates="" Nov 04 03:16:41.537196 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532854 2578 flags.go:64] FLAG: --file-check-frequency="20s" Nov 04 03:16:41.537196 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532857 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 04 03:16:41.537196 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532861 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 04 03:16:41.537196 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532865 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 04 03:16:41.537196 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532868 2578 flags.go:64] FLAG: --healthz-port="10248" Nov 04 03:16:41.537196 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532872 2578 flags.go:64] FLAG: --help="false" Nov 04 03:16:41.537196 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532875 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:16:41.537196 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532878 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 04 03:16:41.537196 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532882 2578 flags.go:64] FLAG: --http-check-frequency="20s" Nov 04 03:16:41.537196 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532885 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 04 03:16:41.537196 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532888 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 04 03:16:41.537196 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532891 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 04 03:16:41.537196 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532894 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 04 03:16:41.537196 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532897 2578 flags.go:64] FLAG: --image-service-endpoint="" Nov 04 03:16:41.537732 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532900 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 04 03:16:41.537732 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532903 2578 flags.go:64] FLAG: --kube-api-burst="100" Nov 04 03:16:41.537732 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532906 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 04 03:16:41.537732 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532909 2578 flags.go:64] FLAG: --kube-api-qps="50" Nov 04 03:16:41.537732 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532913 2578 flags.go:64] FLAG: --kube-reserved="" Nov 04 03:16:41.537732 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532916 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 04 03:16:41.537732 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532918 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 04 03:16:41.537732 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532922 2578 flags.go:64] FLAG: --kubelet-cgroups="" Nov 04 03:16:41.537732 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532924 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 04 03:16:41.537732 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532927 2578 flags.go:64] FLAG: --lock-file="" Nov 04 03:16:41.537732 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532930 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 04 03:16:41.537732 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532933 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 04 03:16:41.537732 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532954 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 04 03:16:41.537732 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532961 2578 flags.go:64] FLAG: --log-json-split-stream="false" Nov 04 03:16:41.537732 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532964 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 04 03:16:41.537732 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532967 2578 flags.go:64] FLAG: --log-text-split-stream="false" Nov 04 03:16:41.537732 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532970 2578 flags.go:64] FLAG: --logging-format="text" Nov 04 03:16:41.537732 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532974 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 04 03:16:41.537732 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532977 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 04 03:16:41.537732 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532980 2578 flags.go:64] FLAG: --manifest-url="" Nov 04 03:16:41.537732 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532984 2578 flags.go:64] FLAG: --manifest-url-header="" Nov 04 03:16:41.537732 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532989 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 04 03:16:41.537732 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532992 2578 flags.go:64] FLAG: --max-open-files="1000000" Nov 04 03:16:41.537732 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532996 2578 flags.go:64] FLAG: --max-pods="110" Nov 04 03:16:41.537732 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.532999 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 04 03:16:41.538357 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533002 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 04 03:16:41.538357 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533005 2578 flags.go:64] FLAG: --memory-manager-policy="None" Nov 04 03:16:41.538357 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533008 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 04 03:16:41.538357 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533011 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 04 03:16:41.538357 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533014 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 04 03:16:41.538357 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533017 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 04 03:16:41.538357 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533026 2578 flags.go:64] FLAG: --node-status-max-images="50" Nov 04 03:16:41.538357 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533029 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 04 03:16:41.538357 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533032 2578 flags.go:64] FLAG: --oom-score-adj="-999" Nov 04 03:16:41.538357 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533035 2578 flags.go:64] FLAG: --pod-cidr="" Nov 04 03:16:41.538357 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533038 2578 flags.go:64] FLAG: --pod-manifest-path="" Nov 04 03:16:41.538357 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533041 2578 flags.go:64] FLAG: --pod-max-pids="-1" Nov 04 03:16:41.538357 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533044 2578 flags.go:64] FLAG: --pods-per-core="0" Nov 04 03:16:41.538357 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533047 2578 flags.go:64] FLAG: --port="10250" Nov 04 03:16:41.538357 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533050 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 04 03:16:41.538357 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533053 2578 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0280834f1c12e1f06" Nov 04 03:16:41.538357 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533056 2578 flags.go:64] FLAG: --qos-reserved="" Nov 04 03:16:41.538357 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533059 2578 flags.go:64] FLAG: --read-only-port="10255" Nov 04 03:16:41.538357 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533062 2578 flags.go:64] FLAG: --register-node="true" Nov 04 03:16:41.538357 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533065 2578 flags.go:64] FLAG: --register-with-taints="" Nov 04 03:16:41.538357 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533069 2578 flags.go:64] FLAG: --registry-burst="10" Nov 04 03:16:41.538357 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533072 2578 flags.go:64] FLAG: --registry-qps="5" Nov 04 03:16:41.538357 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533074 2578 flags.go:64] FLAG: --reserved-cpus="" Nov 04 03:16:41.538357 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533078 2578 flags.go:64] FLAG: --reserved-memory="" Nov 04 03:16:41.538357 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533082 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 04 03:16:41.538357 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533085 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 04 03:16:41.538972 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533088 2578 flags.go:64] FLAG: --rotate-certificates="false" Nov 04 03:16:41.538972 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533092 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 04 03:16:41.538972 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533095 2578 flags.go:64] FLAG: --runonce="false" Nov 04 03:16:41.538972 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533098 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 04 03:16:41.538972 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533108 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 04 03:16:41.538972 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533112 2578 flags.go:64] FLAG: --seccomp-default="false" Nov 04 03:16:41.538972 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533114 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 04 03:16:41.538972 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533117 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 04 03:16:41.538972 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533120 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 04 03:16:41.538972 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533124 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 04 03:16:41.538972 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533127 2578 flags.go:64] FLAG: --storage-driver-password="root" Nov 04 03:16:41.538972 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533130 2578 flags.go:64] FLAG: --storage-driver-secure="false" Nov 04 03:16:41.538972 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533133 2578 flags.go:64] FLAG: --storage-driver-table="stats" Nov 04 03:16:41.538972 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533136 2578 flags.go:64] FLAG: --storage-driver-user="root" Nov 04 03:16:41.538972 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533139 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 04 03:16:41.538972 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533142 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 04 03:16:41.538972 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533145 2578 flags.go:64] FLAG: --system-cgroups="" Nov 04 03:16:41.538972 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533148 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 04 03:16:41.538972 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533153 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 04 03:16:41.538972 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533156 2578 flags.go:64] FLAG: --tls-cert-file="" Nov 04 03:16:41.538972 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.533159 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 04 03:16:41.538972 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.534072 2578 flags.go:64] FLAG: --tls-min-version="" Nov 04 03:16:41.538972 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.534076 2578 flags.go:64] FLAG: --tls-private-key-file="" Nov 04 03:16:41.538972 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.534079 2578 flags.go:64] FLAG: --topology-manager-policy="none" Nov 04 03:16:41.538972 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.534082 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 04 03:16:41.539598 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.534086 2578 flags.go:64] FLAG: --topology-manager-scope="container" Nov 04 03:16:41.539598 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.534089 2578 flags.go:64] FLAG: --v="2" Nov 04 03:16:41.539598 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.534095 2578 flags.go:64] FLAG: --version="false" Nov 04 03:16:41.539598 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.534099 2578 flags.go:64] FLAG: --vmodule="" Nov 04 03:16:41.539598 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.534104 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 04 03:16:41.539598 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.534109 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 04 03:16:41.539598 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534263 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:16:41.539598 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534267 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:16:41.539598 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534275 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:16:41.539598 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534279 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:16:41.539598 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534282 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:16:41.539598 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534285 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:16:41.539598 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534289 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:16:41.539598 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534291 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:16:41.539598 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534294 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:16:41.539598 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534297 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:16:41.539598 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534299 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:16:41.539598 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534302 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:16:41.539598 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534305 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:16:41.539598 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534308 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:16:41.540086 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534310 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:16:41.540086 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534313 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:16:41.540086 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534315 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:16:41.540086 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534318 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:16:41.540086 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534321 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:16:41.540086 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534324 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:16:41.540086 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534326 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:16:41.540086 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534329 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:16:41.540086 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534332 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:16:41.540086 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534334 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:16:41.540086 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534337 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:16:41.540086 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534340 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:16:41.540086 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534342 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:16:41.540086 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534345 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:16:41.540086 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534347 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:16:41.540086 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534350 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:16:41.540086 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534353 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:16:41.540086 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534356 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:16:41.540086 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534359 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:16:41.540086 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534361 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:16:41.540630 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534366 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:16:41.540630 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534368 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:16:41.540630 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534371 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:16:41.540630 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534373 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:16:41.540630 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534376 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:16:41.540630 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534378 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:16:41.540630 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534381 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:16:41.540630 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534383 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:16:41.540630 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534386 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:16:41.540630 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534389 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:16:41.540630 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534391 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:16:41.540630 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534394 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:16:41.540630 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534396 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:16:41.540630 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534399 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:16:41.540630 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534401 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:16:41.540630 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534404 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:16:41.540630 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534407 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:16:41.540630 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534409 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:16:41.540630 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534412 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:16:41.540630 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534414 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:16:41.541135 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534417 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:16:41.541135 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534420 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:16:41.541135 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534422 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:16:41.541135 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534425 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:16:41.541135 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534428 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:16:41.541135 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534430 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:16:41.541135 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534433 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:16:41.541135 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534435 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:16:41.541135 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534438 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:16:41.541135 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534440 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:16:41.541135 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534443 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:16:41.541135 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534446 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:16:41.541135 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534450 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:16:41.541135 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534453 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:16:41.541135 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534455 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:16:41.541135 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534458 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:16:41.541135 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534460 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:16:41.541135 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534463 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:16:41.541135 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534465 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:16:41.541592 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534468 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:16:41.541592 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534471 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:16:41.541592 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534473 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:16:41.541592 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534476 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:16:41.541592 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534478 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:16:41.541592 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534481 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:16:41.541592 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534484 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:16:41.541592 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534486 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:16:41.541592 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534489 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:16:41.541592 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534491 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:16:41.541592 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534494 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:16:41.541592 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534496 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:16:41.541592 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534499 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:16:41.541592 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534502 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:16:41.541592 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534504 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:16:41.541592 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534507 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:16:41.541592 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534509 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:16:41.541592 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534512 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:16:41.541592 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534514 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:16:41.541592 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534517 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:16:41.542278 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534522 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:16:41.542278 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534525 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:16:41.542278 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534528 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:16:41.542278 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534531 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:16:41.542278 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534535 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:16:41.542278 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534540 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:16:41.542278 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534543 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:16:41.542278 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534546 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:16:41.542278 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534548 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:16:41.542278 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534551 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:16:41.542278 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534554 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:16:41.542278 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534556 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:16:41.542278 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534559 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:16:41.542278 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.534588 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 03:16:41.542278 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534629 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:16:41.542867 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534634 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:16:41.542867 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534637 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:16:41.542867 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534640 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:16:41.542867 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534643 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:16:41.542867 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534646 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:16:41.542867 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534649 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:16:41.542867 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534651 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:16:41.542867 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534654 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:16:41.542867 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534657 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:16:41.542867 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534660 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:16:41.542867 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534664 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:16:41.542867 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534668 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:16:41.542867 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534671 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:16:41.542867 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534673 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:16:41.542867 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534678 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:16:41.542867 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534680 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:16:41.542867 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534683 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:16:41.542867 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534686 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:16:41.542867 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534688 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:16:41.543524 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534692 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:16:41.543524 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534694 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:16:41.543524 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534697 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:16:41.543524 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534700 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:16:41.543524 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534704 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:16:41.543524 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534707 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:16:41.543524 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534711 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:16:41.543524 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534713 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:16:41.543524 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534716 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:16:41.543524 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534718 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:16:41.543524 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534721 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:16:41.543524 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534724 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:16:41.543524 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534727 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:16:41.543524 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534730 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:16:41.543524 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534732 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:16:41.543524 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534735 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:16:41.543524 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534737 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:16:41.543524 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534740 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:16:41.543524 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534742 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:16:41.543524 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534745 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:16:41.544041 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534747 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:16:41.544041 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534750 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:16:41.544041 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534753 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:16:41.544041 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534755 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:16:41.544041 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534758 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:16:41.544041 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534760 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:16:41.544041 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534762 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:16:41.544041 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534766 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:16:41.544041 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534769 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:16:41.544041 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534771 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:16:41.544041 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534774 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:16:41.544041 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534776 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:16:41.544041 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534779 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:16:41.544041 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534781 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:16:41.544041 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534784 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:16:41.544041 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534786 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:16:41.544041 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534790 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:16:41.544041 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534794 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:16:41.544041 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534796 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:16:41.544041 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534799 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:16:41.544513 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534802 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:16:41.544513 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534804 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:16:41.544513 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534807 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:16:41.544513 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534809 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:16:41.544513 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534812 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:16:41.544513 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534814 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:16:41.544513 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534817 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:16:41.544513 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534819 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:16:41.544513 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534822 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:16:41.544513 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534824 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:16:41.544513 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534827 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:16:41.544513 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534829 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:16:41.544513 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534832 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:16:41.544513 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534834 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:16:41.544513 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534837 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:16:41.544513 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534839 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:16:41.544513 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534842 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:16:41.544513 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534844 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:16:41.544513 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534847 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:16:41.544513 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534850 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:16:41.545043 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534853 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:16:41.545043 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534855 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:16:41.545043 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534858 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:16:41.545043 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534861 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:16:41.545043 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534863 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:16:41.545043 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534865 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:16:41.545043 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534868 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:16:41.545043 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534870 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:16:41.545043 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534874 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:16:41.545043 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534877 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:16:41.545043 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534879 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:16:41.545043 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534882 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:16:41.545043 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534885 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:16:41.545043 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534887 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:16:41.545043 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534890 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:16:41.545043 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534892 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:16:41.545043 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534895 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:16:41.545043 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534897 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:16:41.545043 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534900 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:16:41.545542 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534902 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:16:41.545542 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534905 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:16:41.545542 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534908 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:16:41.545542 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534912 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:16:41.545542 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534916 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:16:41.545542 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534920 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:16:41.545542 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.534923 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:16:41.545542 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.544207 2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 04 03:16:41.545542 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.544228 2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 04 03:16:41.545542 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.544252 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 04 03:16:41.545542 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.544257 2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 04 03:16:41.545542 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544321 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:16:41.545542 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544328 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:16:41.545542 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544331 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:16:41.545542 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544334 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:16:41.545542 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544337 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:16:41.545542 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544340 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:16:41.545542 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544342 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:16:41.545542 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544346 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:16:41.546004 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544351 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:16:41.546004 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544354 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:16:41.546004 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544357 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:16:41.546004 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544359 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:16:41.546004 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544362 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:16:41.546004 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544365 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:16:41.546004 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544367 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:16:41.546004 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544370 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:16:41.546004 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544373 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:16:41.546004 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544375 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:16:41.546004 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544379 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:16:41.546004 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544383 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:16:41.546004 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544387 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:16:41.546004 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544390 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:16:41.546004 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544392 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:16:41.546004 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544395 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:16:41.546004 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544398 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:16:41.546004 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544400 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:16:41.546004 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544403 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:16:41.546445 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544406 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:16:41.546445 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544409 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:16:41.546445 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544412 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:16:41.546445 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544415 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:16:41.546445 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544418 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:16:41.546445 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544421 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:16:41.546445 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544423 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:16:41.546445 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544426 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:16:41.546445 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544428 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:16:41.546445 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544431 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:16:41.546445 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544433 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:16:41.546445 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544436 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:16:41.546445 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544438 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:16:41.546445 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544441 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:16:41.546445 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544443 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:16:41.546445 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544446 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:16:41.546445 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544449 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:16:41.546445 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544452 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:16:41.546445 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544454 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:16:41.546445 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544457 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:16:41.546906 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544460 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:16:41.546906 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544462 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:16:41.546906 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544464 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:16:41.546906 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544467 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:16:41.546906 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544469 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:16:41.546906 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544472 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:16:41.546906 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544474 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:16:41.546906 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544477 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:16:41.546906 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544480 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:16:41.546906 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544483 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:16:41.546906 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544485 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:16:41.546906 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544487 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:16:41.546906 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544490 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:16:41.546906 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544493 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:16:41.546906 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544497 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:16:41.546906 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544500 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:16:41.546906 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544502 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:16:41.546906 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544505 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:16:41.546906 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544508 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:16:41.546906 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544510 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:16:41.547387 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544513 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:16:41.547387 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544515 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:16:41.547387 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544518 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:16:41.547387 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544521 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:16:41.547387 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544523 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:16:41.547387 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544526 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:16:41.547387 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544529 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:16:41.547387 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544532 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:16:41.547387 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544534 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:16:41.547387 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544537 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:16:41.547387 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544539 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:16:41.547387 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544542 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:16:41.547387 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544545 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:16:41.547387 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544548 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:16:41.547387 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544550 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:16:41.547387 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544553 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:16:41.547387 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544556 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:16:41.547387 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544558 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:16:41.547387 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544561 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:16:41.547830 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544564 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:16:41.547830 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544566 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:16:41.547830 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544569 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:16:41.547830 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544572 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:16:41.547830 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544574 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:16:41.547830 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544577 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:16:41.547830 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544579 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:16:41.547830 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544582 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:16:41.547830 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544585 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:16:41.547830 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544587 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:16:41.547830 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544590 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:16:41.547830 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544592 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:16:41.547830 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544595 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:16:41.547830 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544597 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:16:41.547830 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544600 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:16:41.547830 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544602 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:16:41.547830 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544605 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:16:41.547830 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544607 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:16:41.547830 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544610 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:16:41.547830 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544612 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:16:41.548350 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.544644 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 03:16:41.548350 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544788 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:16:41.548350 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544793 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:16:41.548350 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544796 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:16:41.548350 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544799 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:16:41.548350 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544802 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:16:41.548350 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544806 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:16:41.548350 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544809 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:16:41.548350 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544812 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:16:41.548350 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544814 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:16:41.548350 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544817 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:16:41.548350 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544819 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:16:41.548350 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544822 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:16:41.548350 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544824 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:16:41.548350 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544827 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:16:41.548350 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544830 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:16:41.548742 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544832 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:16:41.548742 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544835 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:16:41.548742 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544838 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:16:41.548742 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544841 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:16:41.548742 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544843 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:16:41.548742 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544846 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:16:41.548742 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544849 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:16:41.548742 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544851 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:16:41.548742 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544854 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:16:41.548742 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544856 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:16:41.548742 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544859 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:16:41.548742 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544861 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:16:41.548742 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544864 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:16:41.548742 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544867 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:16:41.548742 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544869 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:16:41.548742 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544871 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:16:41.548742 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544874 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:16:41.548742 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544877 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:16:41.548742 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544879 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:16:41.549224 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544881 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:16:41.549224 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544884 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:16:41.549224 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544887 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:16:41.549224 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544889 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:16:41.549224 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544892 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:16:41.549224 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544895 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:16:41.549224 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544898 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:16:41.549224 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544900 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:16:41.549224 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544903 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:16:41.549224 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544906 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:16:41.549224 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544908 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:16:41.549224 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544911 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:16:41.549224 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544914 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:16:41.549224 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544917 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:16:41.549224 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544919 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:16:41.549224 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544922 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:16:41.549224 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544925 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:16:41.549224 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544927 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:16:41.549224 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544930 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:16:41.549224 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544932 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:16:41.549692 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544952 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:16:41.549692 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544955 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:16:41.549692 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544957 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:16:41.549692 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544960 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:16:41.549692 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544963 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:16:41.549692 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544965 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:16:41.549692 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544968 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:16:41.549692 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544970 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:16:41.549692 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544973 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:16:41.549692 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544976 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:16:41.549692 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544979 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:16:41.549692 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544981 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:16:41.549692 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544984 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:16:41.549692 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544986 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:16:41.549692 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544989 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:16:41.549692 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544992 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:16:41.549692 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544994 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:16:41.549692 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.544997 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:16:41.549692 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545000 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:16:41.549692 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545002 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:16:41.550178 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545005 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:16:41.550178 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545007 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:16:41.550178 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545010 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:16:41.550178 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545013 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:16:41.550178 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545016 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:16:41.550178 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545020 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:16:41.550178 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545023 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:16:41.550178 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545026 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:16:41.550178 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545028 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:16:41.550178 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545031 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:16:41.550178 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545033 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:16:41.550178 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545036 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:16:41.550178 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545038 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:16:41.550178 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545041 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:16:41.550178 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545044 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:16:41.550178 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545047 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:16:41.550178 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545050 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:16:41.550178 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545052 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:16:41.550178 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545056 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:16:41.550616 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545059 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:16:41.550616 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545061 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:16:41.550616 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545064 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:16:41.550616 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545068 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:16:41.550616 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545072 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:16:41.550616 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545075 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:16:41.550616 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545077 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:16:41.550616 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545080 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:16:41.550616 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545083 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:16:41.550616 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545085 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:16:41.550616 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545088 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:16:41.550616 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545094 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:16:41.550616 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545097 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:16:41.550616 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.545129 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 03:16:41.550616 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545164 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:16:41.551013 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545168 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:16:41.551013 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545171 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:16:41.551013 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545174 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:16:41.551013 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545177 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:16:41.551013 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545179 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:16:41.551013 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545182 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:16:41.551013 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545185 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:16:41.551013 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545187 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:16:41.551013 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545190 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:16:41.551013 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545193 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:16:41.551013 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545195 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:16:41.551013 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545198 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:16:41.551013 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545200 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:16:41.551013 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545203 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:16:41.551013 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545205 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:16:41.551013 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545208 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:16:41.551013 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545210 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:16:41.551013 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545214 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:16:41.551013 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545217 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:16:41.551460 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545221 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:16:41.551460 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545224 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:16:41.551460 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545226 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:16:41.551460 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545229 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:16:41.551460 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545231 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:16:41.551460 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545234 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:16:41.551460 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545237 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:16:41.551460 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545239 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:16:41.551460 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545243 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:16:41.551460 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545247 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:16:41.551460 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545251 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:16:41.551460 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545254 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:16:41.551460 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545257 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:16:41.551460 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545259 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:16:41.551460 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545262 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:16:41.551460 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545264 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:16:41.551460 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545268 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:16:41.551460 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545272 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:16:41.551460 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545275 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:16:41.551919 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545277 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:16:41.551919 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545280 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:16:41.551919 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545283 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:16:41.551919 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545285 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:16:41.551919 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545288 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:16:41.551919 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545291 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:16:41.551919 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545293 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:16:41.551919 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545296 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:16:41.551919 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545299 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:16:41.551919 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545301 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:16:41.551919 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545304 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:16:41.551919 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545306 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:16:41.551919 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545309 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:16:41.551919 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545312 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:16:41.551919 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545315 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:16:41.551919 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545317 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:16:41.551919 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545320 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:16:41.551919 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545322 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:16:41.551919 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545324 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:16:41.551919 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545327 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:16:41.552402 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545329 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:16:41.552402 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545332 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:16:41.552402 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545336 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:16:41.552402 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545338 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:16:41.552402 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545341 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:16:41.552402 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545344 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:16:41.552402 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545346 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:16:41.552402 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545349 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:16:41.552402 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545351 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:16:41.552402 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545354 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:16:41.552402 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545356 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:16:41.552402 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545359 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:16:41.552402 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545362 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:16:41.552402 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545364 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:16:41.552402 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545366 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:16:41.552402 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545369 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:16:41.552402 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545371 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:16:41.552402 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545374 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:16:41.552402 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545376 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:16:41.552840 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545379 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:16:41.552840 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545381 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:16:41.552840 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545384 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:16:41.552840 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545386 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:16:41.552840 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545389 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:16:41.552840 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545392 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:16:41.552840 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545395 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:16:41.552840 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545397 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:16:41.552840 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545400 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:16:41.552840 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545402 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:16:41.552840 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545404 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:16:41.552840 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545407 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:16:41.552840 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545410 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:16:41.552840 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545412 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:16:41.552840 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545415 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:16:41.552840 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545419 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:16:41.552840 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545421 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:16:41.552840 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545424 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:16:41.552840 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545427 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:16:41.552840 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545430 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:16:41.553370 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545432 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:16:41.553370 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545435 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:16:41.553370 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545437 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:16:41.553370 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545440 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:16:41.553370 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545443 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:16:41.553370 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545446 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:16:41.553370 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545448 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:16:41.553370 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:41.545451 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:16:41.553370 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.545519 2578 server.go:957] "Client rotation is on, will bootstrap in background" Nov 04 03:16:41.553370 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.547891 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 04 03:16:41.553370 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.548825 2578 server.go:1014] "Starting client certificate rotation" Nov 04 03:16:41.553370 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.548927 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 03:16:41.553370 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.549922 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 03:16:41.553370 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.550989 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 03:16:41.556852 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.556833 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 03:16:41.564725 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.564704 2578 log.go:25] "Validated CRI v1 runtime API" Nov 04 03:16:41.568701 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.568683 2578 log.go:25] "Validated CRI v1 image API" Nov 04 03:16:41.569593 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.569569 2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 04 03:16:41.573438 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.573409 2578 fs.go:135] Filesystem UUIDs: map[2acf8da6-c497-48ca-9887-7c0f8c314692:/dev/nvme0n1p3 732e0981-1f45-46c0-8881-cb4c01bb118c:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 04 03:16:41.573516 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.573436 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 04 03:16:41.579252 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.579114 2578 manager.go:217] Machine: {Timestamp:2026-11-04 03:16:41.57785335 +0000 UTC m=+0.358082010 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100137 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24c41d1b2460027f6ac682d0506f92 SystemUUID:ec24c41d-1b24-6002-7f6a-c682d0506f92 BootID:0243aa05-44b8-47e3-9575-cd1be2423faf Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:30:22:f9:d6:4d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:30:22:f9:d6:4d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0e:4f:b3:2b:63:e7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 04 03:16:41.579887 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.579872 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 04 03:16:41.580036 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.580012 2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 04 03:16:41.581244 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.581212 2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 04 03:16:41.581394 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.581246 2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-6.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 04 03:16:41.581445 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.581400 2578 topology_manager.go:143] "Creating topology manager with none policy" Nov 04 03:16:41.581445 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.581409 2578 container_manager_linux.go:311] "Creating device plugin manager" Nov 04 03:16:41.581499 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.581444 2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 03:16:41.581499 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.581472 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 03:16:41.582757 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.582745 2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 04 03:16:41.587204 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.587187 2578 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 04 03:16:41.587362 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.587353 2578 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 04 03:16:41.588172 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.588160 2578 kubelet.go:493] "Attempting to sync node with API server" Nov 04 03:16:41.588211 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.588183 2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 04 03:16:41.588211 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.588195 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 04 03:16:41.588277 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.588233 2578 kubelet.go:405] "Adding apiserver pod source" Nov 04 03:16:41.588277 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.588247 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 04 03:16:41.591331 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.591302 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 03:16:41.594138 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.594117 2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 04 03:16:41.601356 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.601338 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 03:16:41.602369 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.602357 2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 04 03:16:41.602404 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.602388 2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 04 03:16:41.602557 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.602546 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 04 03:16:41.602591 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.602561 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 04 03:16:41.602591 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.602567 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 04 03:16:41.602591 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.602573 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 04 03:16:41.602591 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.602579 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 04 03:16:41.602591 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.602585 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 04 03:16:41.603202 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.603193 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 04 03:16:41.603237 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.603203 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 04 03:16:41.603237 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.603211 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 04 03:16:41.603237 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.603217 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 04 03:16:41.603237 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.603225 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 04 03:16:41.603237 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.603236 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 04 03:16:41.603921 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.603911 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 04 03:16:41.603921 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.603921 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 04 03:16:41.607859 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.607843 2578 server.go:1263] "Started kubelet" Nov 04 03:16:41.608013 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.607968 2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 04 03:16:41.608068 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.607995 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 04 03:16:41.608101 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.608075 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 04 03:16:41.609795 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.609723 2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 04 03:16:41.610100 ip-10-0-133-6 systemd[1]: Started Kubernetes Kubelet. Nov 04 03:16:41.612758 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.612743 2578 server.go:317] "Adding debug handlers to kubelet server" Nov 04 03:16:41.614598 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:41.614569 2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 04 03:16:41.616714 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.616695 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 04 03:16:41.617311 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.617281 2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 04 03:16:41.617457 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:41.617438 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-6.us-west-1.compute.internal\" not found" Nov 04 03:16:41.618129 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.618085 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 04 03:16:41.618129 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.618111 2578 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 04 03:16:41.618290 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.618141 2578 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 04 03:16:41.618290 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.618222 2578 reconstruct.go:97] "Volume reconstruction finished" Nov 04 03:16:41.618290 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.618230 2578 reconciler.go:29] "Reconciler: start to sync state" Nov 04 03:16:41.618665 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.618652 2578 factory.go:153] Registering CRI-O factory Nov 04 03:16:41.618743 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.618670 2578 factory.go:223] Registration of the crio container factory successfully Nov 04 03:16:41.618743 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.618709 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 04 03:16:41.618743 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.618716 2578 factory.go:55] Registering systemd factory Nov 04 03:16:41.618743 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.618721 2578 factory.go:223] Registration of the systemd container factory successfully Nov 04 03:16:41.618743 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.618735 2578 factory.go:103] Registering Raw factory Nov 04 03:16:41.618743 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.618744 2578 manager.go:1199] Started watching for new ooms in manager Nov 04 03:16:41.619680 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.619666 2578 manager.go:319] Starting recovery of all containers Nov 04 03:16:41.632617 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.632597 2578 manager.go:324] Recovery completed Nov 04 03:16:41.636727 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.636707 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 03:16:41.638961 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.638920 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-6.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 03:16:41.639024 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.638984 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-6.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 03:16:41.639024 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.638996 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-6.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 03:16:41.639502 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.639489 2578 cpu_manager.go:226] "Starting" policy="none" Nov 04 03:16:41.639558 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.639501 2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 04 03:16:41.639558 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.639514 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 03:16:41.639652 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.639566 2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 04 03:16:41.640216 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.640199 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-6.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 03:16:41.640299 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.640232 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-6.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 03:16:41.640299 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.640247 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-6.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 03:16:41.641726 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.641712 2578 policy_none.go:50] "Start" Nov 04 03:16:41.641804 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.641730 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 04 03:16:41.641804 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.641743 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 04 03:16:41.643824 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.643810 2578 policy_none.go:44] "Start" Nov 04 03:16:41.679818 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.679621 2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 04 03:16:41.683982 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.683968 2578 manager.go:341] "Starting Device Plugin manager" Nov 04 03:16:41.684041 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:41.684025 2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 04 03:16:41.684041 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.684039 2578 server.go:87] "Starting device plugin registration server" Nov 04 03:16:41.684291 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.684282 2578 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 04 03:16:41.684322 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.684293 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 04 03:16:41.684408 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.684390 2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 04 03:16:41.684506 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.684485 2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 04 03:16:41.684506 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.684496 2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 04 03:16:41.685173 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:41.685154 2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 04 03:16:41.685256 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:41.685217 2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-6.us-west-1.compute.internal\" not found" Nov 04 03:16:41.745145 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.745107 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 04 03:16:41.747158 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.746650 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 04 03:16:41.747158 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.746674 2578 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 04 03:16:41.747158 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.746699 2578 kubelet.go:2521] "Starting kubelet main sync loop" Nov 04 03:16:41.747158 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:41.746733 2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 04 03:16:41.750893 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.750864 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-6.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 04 03:16:41.750893 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:41.750864 2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-6.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 04 03:16:41.751828 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:41.750864 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-6.us-west-1.compute.internal.18e4b937f65b32cb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-6.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-6.us-west-1.compute.internal,},FirstTimestamp:2026-11-04 03:16:41.607811787 +0000 UTC m=+0.388040447,LastTimestamp:2026-11-04 03:16:41.607811787 +0000 UTC m=+0.388040447,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-6.us-west-1.compute.internal,}" Nov 04 03:16:41.785401 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.785328 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 03:16:41.786444 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.786426 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-6.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 03:16:41.786526 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.786460 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-6.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 03:16:41.786526 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.786470 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-6.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 03:16:41.786526 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.786493 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:16:41.803859 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:41.803817 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 04 03:16:41.804008 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:41.803840 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-6.us-west-1.compute.internal.18e4b937f8368a39 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-6.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-6.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-6.us-west-1.compute.internal,},FirstTimestamp:2026-11-04 03:16:41.638963769 +0000 UTC m=+0.419192436,LastTimestamp:2026-11-04 03:16:41.638963769 +0000 UTC m=+0.419192436,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-6.us-west-1.compute.internal,}" Nov 04 03:16:41.831346 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.831318 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 04 03:16:41.831463 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.831380 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g98dj" Nov 04 03:16:41.847451 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.847379 2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-6.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal"] Nov 04 03:16:41.847538 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.847490 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 03:16:41.848675 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.848655 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-6.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 03:16:41.848777 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.848693 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-6.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 03:16:41.848777 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.848708 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-6.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 03:16:41.851055 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.851038 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 03:16:41.851654 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.851626 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:16:41.851765 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.851670 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 03:16:41.851861 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.851844 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-6.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 03:16:41.851899 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.851877 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-6.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 03:16:41.851899 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.851888 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-6.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 03:16:41.852417 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.852401 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-6.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 03:16:41.852501 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.852429 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-6.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 03:16:41.852501 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.852443 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-6.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 03:16:41.854018 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.854002 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:16:41.854096 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.854029 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 03:16:41.854698 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.854681 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-6.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 03:16:41.854787 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.854707 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-6.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 03:16:41.854787 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.854718 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-6.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 03:16:41.878991 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:41.878949 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-6.us-west-1.compute.internal\" not found" node="ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:16:41.885588 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:41.885565 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-6.us-west-1.compute.internal\" not found" node="ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:16:41.910244 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.910210 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 03:16:41.911634 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:41.911619 2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 03:16:42.011867 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.011829 2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:16:42.012068 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.011927 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 03:16:42.018118 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.018095 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:16:42.019237 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.019218 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ca990c51a9c4be7e8fb143b232789941-config\") pod \"kube-apiserver-proxy-ip-10-0-133-6.us-west-1.compute.internal\" (UID: \"ca990c51a9c4be7e8fb143b232789941\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:16:42.019286 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.019247 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/12436f4d0565699f03041c74b80c999a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal\" (UID: \"12436f4d0565699f03041c74b80c999a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:16:42.019286 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.019264 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12436f4d0565699f03041c74b80c999a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal\" (UID: \"12436f4d0565699f03041c74b80c999a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:16:42.076801 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.076780 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 03:16:42.077559 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.077541 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:16:42.119406 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.119381 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ca990c51a9c4be7e8fb143b232789941-config\") pod \"kube-apiserver-proxy-ip-10-0-133-6.us-west-1.compute.internal\" (UID: \"ca990c51a9c4be7e8fb143b232789941\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:16:42.119500 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.119419 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/12436f4d0565699f03041c74b80c999a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal\" (UID: \"12436f4d0565699f03041c74b80c999a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:16:42.119500 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.119445 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12436f4d0565699f03041c74b80c999a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal\" (UID: \"12436f4d0565699f03041c74b80c999a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:16:42.119500 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.119462 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ca990c51a9c4be7e8fb143b232789941-config\") pod \"kube-apiserver-proxy-ip-10-0-133-6.us-west-1.compute.internal\" (UID: \"ca990c51a9c4be7e8fb143b232789941\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:16:42.119500 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.119483 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12436f4d0565699f03041c74b80c999a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal\" (UID: \"12436f4d0565699f03041c74b80c999a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:16:42.119618 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.119522 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/12436f4d0565699f03041c74b80c999a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal\" (UID: \"12436f4d0565699f03041c74b80c999a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:16:42.136642 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.136615 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 03:16:42.181189 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.181163 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:16:42.188713 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.188694 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:16:42.548534 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.548425 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 04 03:16:42.549057 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.548621 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 04 03:16:42.549057 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.548634 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 04 03:16:42.549057 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:42.548692 2578 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a0eed642b5924453b9c4f38476315c56-78f2aa2176e6b7a4.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-133-6.us-west-1.compute.internal.18e4b937f8371857\": read tcp 10.0.133.6:49584->18.220.178.215:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-133-6.us-west-1.compute.internal.18e4b937f8371857 default 5497 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-6.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-6.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-6.us-west-1.compute.internal,},FirstTimestamp:2026-11-04 03:16:41 +0000 UTC,LastTimestamp:2026-11-04 03:16:41.78647423 +0000 UTC m=+0.566702890,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-6.us-west-1.compute.internal,}" Nov 04 03:16:42.589735 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.589703 2578 apiserver.go:52] "Watching apiserver" Nov 04 03:16:42.617818 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.617789 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 04 03:16:42.835313 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.835258 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-03 03:11:42 +0000 UTC" deadline="2028-06-29 18:12:23.718747476 +0000 UTC" Nov 04 03:16:42.835313 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.835309 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14486h55m40.883441574s" Nov 04 03:16:42.857293 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.857262 2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 04 03:16:42.857597 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.857577 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-8rmjv","openshift-dns/node-resolver-r9xhm","openshift-multus/multus-t2twg","openshift-ovn-kubernetes/ovnkube-node-bqm4b","kube-system/kube-apiserver-proxy-ip-10-0-133-6.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb","openshift-cluster-node-tuning-operator/tuned-7qqx5","openshift-multus/network-metrics-daemon-dzg2c","openshift-image-registry/node-ca-wj6dr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal","kube-system/konnectivity-agent-pmv6n","openshift-multus/multus-additional-cni-plugins-q5p48","openshift-network-diagnostics/network-check-target-2hjpn","openshift-network-operator/iptables-alerter-s2zcf"] Nov 04 03:16:42.859972 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.859953 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:16:42.860046 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:42.860029 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rmjv" podUID="834b5587-1820-4089-a925-ab992fe654b4" Nov 04 03:16:42.861919 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.861906 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r9xhm" Nov 04 03:16:42.863902 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.863879 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t2twg" Nov 04 03:16:42.866031 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.866012 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:42.868131 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.868113 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:16:42.870273 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.870258 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:42.872387 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.872371 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:16:42.872468 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:42.872447 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzg2c" podUID="1aa3f04c-b684-4426-8d46-a36d8c4e77be" Nov 04 03:16:42.874519 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.874497 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wj6dr" Nov 04 03:16:42.876743 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.876724 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pmv6n" Nov 04 03:16:42.878979 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.878963 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:16:42.881196 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.881178 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:16:42.881284 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:42.881260 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hjpn" podUID="719911b7-0c11-4c79-bb86-fdc043d9f9ab" Nov 04 03:16:42.883447 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.883428 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s2zcf" Nov 04 03:16:42.915479 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.915441 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 04 03:16:42.915479 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.915442 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 04 03:16:42.915708 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.915538 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9c548\"" Nov 04 03:16:42.915841 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.915824 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 04 03:16:42.915932 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.915914 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-cj8mj\"" Nov 04 03:16:42.915932 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.915926 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 04 03:16:42.916041 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.915920 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 04 03:16:42.916041 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.915954 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 04 03:16:42.917691 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.917677 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 04 03:16:42.918028 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.918011 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 04 03:16:42.918140 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.918038 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 04 03:16:42.918140 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.918046 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 04 03:16:42.918140 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.918074 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 04 03:16:42.918274 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.918262 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-s72nt\"" Nov 04 03:16:42.918596 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.918583 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 04 03:16:42.919461 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.919444 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 04 03:16:42.919837 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.919778 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 04 03:16:42.920093 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.920068 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 04 03:16:42.920093 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.920074 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 04 03:16:42.920252 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.920151 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5q5qr\"" Nov 04 03:16:42.924672 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.924654 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-host-run-k8s-cni-cncf-io\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:42.924777 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.924683 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-host-run-netns\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:42.924777 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.924709 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmxc6\" (UniqueName: \"kubernetes.io/projected/e5e3933f-6610-46a7-934a-8ce205da98d9-kube-api-access-zmxc6\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:16:42.924777 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.924733 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q24vc\" (UniqueName: \"kubernetes.io/projected/ca5887c7-01d6-44ee-ac30-8b4374663d3e-kube-api-access-q24vc\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:42.924777 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.924757 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/326a1269-c427-4470-86a6-70ee348c8791-ovnkube-config\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:42.924969 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.924778 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-host-var-lib-kubelet\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:42.924969 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.924804 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qzd8\" (UniqueName: \"kubernetes.io/projected/326a1269-c427-4470-86a6-70ee348c8791-kube-api-access-8qzd8\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:42.924969 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.924828 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-cnibin\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:42.924969 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.924877 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqc8z\" (UniqueName: \"kubernetes.io/projected/ca254b76-1d53-44aa-81e5-e40a81265145-kube-api-access-fqc8z\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:42.924969 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.924904 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e5e3933f-6610-46a7-934a-8ce205da98d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:16:42.925222 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.924968 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-var-lib-kubelet\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:42.925222 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.924998 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/326a1269-c427-4470-86a6-70ee348c8791-ovn-node-metrics-cert\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:42.925222 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925031 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-multus-socket-dir-parent\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:42.925222 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925061 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-hostroot\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:42.925222 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925095 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-cnibin\") pod \"multus-additional-cni-plugins-q5p48\" (UID: \"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab\") " pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:16:42.925222 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925124 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-slash\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:42.925222 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925141 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/326a1269-c427-4470-86a6-70ee348c8791-env-overrides\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:42.925222 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925156 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-system-cni-dir\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:42.925222 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925173 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-host-run-multus-certs\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:42.925222 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925187 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-systemd\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:42.925222 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925201 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-sys\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:42.925222 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925216 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-tuned\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:42.925634 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925248 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-system-cni-dir\") pod \"multus-additional-cni-plugins-q5p48\" (UID: \"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab\") " pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:16:42.925634 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925264 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-cni-binary-copy\") pod \"multus-additional-cni-plugins-q5p48\" (UID: \"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab\") " pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:16:42.925634 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925283 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/834b5587-1820-4089-a925-ab992fe654b4-original-pull-secret\") pod \"global-pull-secret-syncer-8rmjv\" (UID: \"834b5587-1820-4089-a925-ab992fe654b4\") " pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:16:42.925634 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925312 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-run-systemd\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:42.925634 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925337 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-node-log\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:42.925634 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925370 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjwjp\" (UniqueName: \"kubernetes.io/projected/1f6b477e-a0fd-49c8-896d-c268a6c06d3f-kube-api-access-vjwjp\") pod \"node-ca-wj6dr\" (UID: \"1f6b477e-a0fd-49c8-896d-c268a6c06d3f\") " pod="openshift-image-registry/node-ca-wj6dr" Nov 04 03:16:42.925634 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925400 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/923f8598-9df4-46bb-8da0-73442c97522f-agent-certs\") pod \"konnectivity-agent-pmv6n\" (UID: \"923f8598-9df4-46bb-8da0-73442c97522f\") " pod="kube-system/konnectivity-agent-pmv6n" Nov 04 03:16:42.925634 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925422 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-multus-cni-dir\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:42.925634 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925447 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q5p48\" (UID: \"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab\") " pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:16:42.925634 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925469 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-run-ovn\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:42.925634 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925483 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-etc-kubernetes\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:42.925634 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925497 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e5e3933f-6610-46a7-934a-8ce205da98d9-sys-fs\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:16:42.925634 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925512 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-run\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:42.925634 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925527 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngsgw\" (UniqueName: \"kubernetes.io/projected/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-kube-api-access-ngsgw\") pod \"multus-additional-cni-plugins-q5p48\" (UID: \"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab\") " pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:16:42.925634 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925542 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-kubelet\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:42.925634 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925555 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-log-socket\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:42.926362 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925568 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-host-var-lib-cni-multus\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:42.926362 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925594 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/834b5587-1820-4089-a925-ab992fe654b4-kubelet-config\") pod \"global-pull-secret-syncer-8rmjv\" (UID: \"834b5587-1820-4089-a925-ab992fe654b4\") " pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:16:42.926362 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925611 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/834b5587-1820-4089-a925-ab992fe654b4-dbus\") pod \"global-pull-secret-syncer-8rmjv\" (UID: \"834b5587-1820-4089-a925-ab992fe654b4\") " pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:16:42.926362 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925625 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdtwq\" (UniqueName: \"kubernetes.io/projected/fb15c957-99e8-4c15-b382-788025661809-kube-api-access-cdtwq\") pod \"node-resolver-r9xhm\" (UID: \"fb15c957-99e8-4c15-b382-788025661809\") " pod="openshift-dns/node-resolver-r9xhm" Nov 04 03:16:42.926362 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925638 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-run-netns\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:42.926362 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925652 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-run-openvswitch\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:42.926362 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925685 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-host-var-lib-cni-bin\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:42.926362 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925716 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-lib-modules\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:42.926362 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925740 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca5887c7-01d6-44ee-ac30-8b4374663d3e-tmp\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:42.926362 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925766 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fb15c957-99e8-4c15-b382-788025661809-hosts-file\") pod \"node-resolver-r9xhm\" (UID: \"fb15c957-99e8-4c15-b382-788025661809\") " pod="openshift-dns/node-resolver-r9xhm" Nov 04 03:16:42.926362 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925787 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb15c957-99e8-4c15-b382-788025661809-tmp-dir\") pod \"node-resolver-r9xhm\" (UID: \"fb15c957-99e8-4c15-b382-788025661809\") " pod="openshift-dns/node-resolver-r9xhm" Nov 04 03:16:42.926362 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925810 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-cni-bin\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:42.926362 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925834 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-cni-netd\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:42.926362 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925863 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:42.926362 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925897 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1f6b477e-a0fd-49c8-896d-c268a6c06d3f-serviceca\") pod \"node-ca-wj6dr\" (UID: \"1f6b477e-a0fd-49c8-896d-c268a6c06d3f\") " pod="openshift-image-registry/node-ca-wj6dr" Nov 04 03:16:42.926362 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925926 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9f7d1536-9b78-4f13-ae4e-349db78d213e-iptables-alerter-script\") pod \"iptables-alerter-s2zcf\" (UID: \"9f7d1536-9b78-4f13-ae4e-349db78d213e\") " pod="openshift-network-operator/iptables-alerter-s2zcf" Nov 04 03:16:42.926993 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.925982 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-run-ovn-kubernetes\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:42.926993 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.926006 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcw55\" (UniqueName: \"kubernetes.io/projected/9f7d1536-9b78-4f13-ae4e-349db78d213e-kube-api-access-kcw55\") pod \"iptables-alerter-s2zcf\" (UID: \"9f7d1536-9b78-4f13-ae4e-349db78d213e\") " pod="openshift-network-operator/iptables-alerter-s2zcf" Nov 04 03:16:42.926993 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.926020 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ca254b76-1d53-44aa-81e5-e40a81265145-multus-daemon-config\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:42.926993 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.926033 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e5e3933f-6610-46a7-934a-8ce205da98d9-socket-dir\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:16:42.926993 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.926049 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-modprobe-d\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:42.926993 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.926080 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-var-lib-openvswitch\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:42.926993 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.926108 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-etc-openvswitch\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:42.926993 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.926134 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-multus-conf-dir\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:42.926993 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.926162 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q5p48\" (UID: \"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab\") " pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:16:42.926993 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.926188 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-systemd-units\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:42.926993 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.926211 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7xg7\" (UniqueName: \"kubernetes.io/projected/719911b7-0c11-4c79-bb86-fdc043d9f9ab-kube-api-access-n7xg7\") pod \"network-check-target-2hjpn\" (UID: \"719911b7-0c11-4c79-bb86-fdc043d9f9ab\") " pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:16:42.926993 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.926233 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9f7d1536-9b78-4f13-ae4e-349db78d213e-host-slash\") pod \"iptables-alerter-s2zcf\" (UID: \"9f7d1536-9b78-4f13-ae4e-349db78d213e\") " pod="openshift-network-operator/iptables-alerter-s2zcf" Nov 04 03:16:42.926993 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.926258 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e5e3933f-6610-46a7-934a-8ce205da98d9-registration-dir\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:16:42.926993 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.926279 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-sysctl-conf\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:42.926993 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.926302 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aa3f04c-b684-4426-8d46-a36d8c4e77be-metrics-certs\") pod \"network-metrics-daemon-dzg2c\" (UID: \"1aa3f04c-b684-4426-8d46-a36d8c4e77be\") " pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:16:42.926993 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.926324 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/923f8598-9df4-46bb-8da0-73442c97522f-konnectivity-ca\") pod \"konnectivity-agent-pmv6n\" (UID: \"923f8598-9df4-46bb-8da0-73442c97522f\") " pod="kube-system/konnectivity-agent-pmv6n" Nov 04 03:16:42.927414 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.926347 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e5e3933f-6610-46a7-934a-8ce205da98d9-device-dir\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:16:42.927414 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.926368 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/326a1269-c427-4470-86a6-70ee348c8791-ovnkube-script-lib\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:42.927414 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.926391 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdrr8\" (UniqueName: \"kubernetes.io/projected/1aa3f04c-b684-4426-8d46-a36d8c4e77be-kube-api-access-zdrr8\") pod \"network-metrics-daemon-dzg2c\" (UID: \"1aa3f04c-b684-4426-8d46-a36d8c4e77be\") " pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:16:42.927414 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.926414 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e5e3933f-6610-46a7-934a-8ce205da98d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:16:42.927414 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.926455 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q5p48\" (UID: \"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab\") " pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:16:42.927414 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.926492 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca254b76-1d53-44aa-81e5-e40a81265145-cni-binary-copy\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:42.927414 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.926520 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-sysconfig\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:42.927414 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.926550 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-kubernetes\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:42.927414 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.926572 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-sysctl-d\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:42.927414 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.926595 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-host\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:42.927414 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.926618 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f6b477e-a0fd-49c8-896d-c268a6c06d3f-host\") pod \"node-ca-wj6dr\" (UID: \"1f6b477e-a0fd-49c8-896d-c268a6c06d3f\") " pod="openshift-image-registry/node-ca-wj6dr" Nov 04 03:16:42.928895 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.928878 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 04 03:16:42.959258 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.959219 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 04 03:16:42.959258 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.959225 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 04 03:16:42.959258 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.959251 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 04 03:16:42.959550 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.959256 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 04 03:16:42.959550 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.959324 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ztc7x\"" Nov 04 03:16:42.959550 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.959403 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qhl58\"" Nov 04 03:16:42.959550 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.959449 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-8nk47\"" Nov 04 03:16:42.961500 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.961484 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 04 03:16:42.961593 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:42.961577 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 04 03:16:44.195097 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.195054 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 04 03:16:44.195257 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.195124 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tcnmd" Nov 04 03:16:44.197676 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.197649 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 04 03:16:44.197676 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.197659 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 04 03:16:44.197812 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.197659 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 04 03:16:44.197857 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.197832 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 04 03:16:44.197906 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.197855 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 04 03:16:44.197906 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.197873 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7jn7l\"" Nov 04 03:16:44.197979 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.197916 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5pzz6\"" Nov 04 03:16:44.209677 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.209649 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qzd8\" (UniqueName: \"kubernetes.io/projected/326a1269-c427-4470-86a6-70ee348c8791-kube-api-access-8qzd8\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.209677 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.209679 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-cnibin\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:44.209836 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.209696 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqc8z\" (UniqueName: \"kubernetes.io/projected/ca254b76-1d53-44aa-81e5-e40a81265145-kube-api-access-fqc8z\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:44.209836 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.209713 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e5e3933f-6610-46a7-934a-8ce205da98d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:16:44.209836 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.209730 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-var-lib-kubelet\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:44.209836 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.209758 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/326a1269-c427-4470-86a6-70ee348c8791-ovn-node-metrics-cert\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.209836 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.209782 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-multus-socket-dir-parent\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:44.209836 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.209803 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-hostroot\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:44.209836 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.209827 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-cnibin\") pod \"multus-additional-cni-plugins-q5p48\" (UID: \"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab\") " pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:16:44.210176 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.209842 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-var-lib-kubelet\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:44.210176 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.209874 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-slash\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.210176 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.209882 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e5e3933f-6610-46a7-934a-8ce205da98d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:16:44.210176 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.209896 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/326a1269-c427-4470-86a6-70ee348c8791-env-overrides\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.210176 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.209803 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-cnibin\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:44.210176 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.209922 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-multus-socket-dir-parent\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:44.210176 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.209938 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-slash\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.210176 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.209941 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-hostroot\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:44.210176 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.209971 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-system-cni-dir\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:44.210176 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.209983 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-cnibin\") pod \"multus-additional-cni-plugins-q5p48\" (UID: \"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab\") " pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:16:44.210176 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.209999 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-host-run-multus-certs\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:44.210176 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.210006 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-system-cni-dir\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:44.210176 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.210024 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-systemd\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:44.210176 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.210027 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-host-run-multus-certs\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:44.210176 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.210047 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-sys\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:44.210176 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.210066 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-systemd\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:44.210176 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.210069 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-tuned\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:44.210176 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.210103 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-sys\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:44.210875 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.210284 2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 04 03:16:44.210875 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.210532 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/326a1269-c427-4470-86a6-70ee348c8791-env-overrides\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.210875 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.210666 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-system-cni-dir\") pod \"multus-additional-cni-plugins-q5p48\" (UID: \"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab\") " pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:16:44.210875 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.210687 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-cni-binary-copy\") pod \"multus-additional-cni-plugins-q5p48\" (UID: \"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab\") " pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:16:44.210875 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.210704 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/834b5587-1820-4089-a925-ab992fe654b4-original-pull-secret\") pod \"global-pull-secret-syncer-8rmjv\" (UID: \"834b5587-1820-4089-a925-ab992fe654b4\") " pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:16:44.210875 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.210726 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-run-systemd\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.210875 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.210744 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-system-cni-dir\") pod \"multus-additional-cni-plugins-q5p48\" (UID: \"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab\") " pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:16:44.210875 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.210784 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-node-log\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.210875 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.210809 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjwjp\" (UniqueName: \"kubernetes.io/projected/1f6b477e-a0fd-49c8-896d-c268a6c06d3f-kube-api-access-vjwjp\") pod \"node-ca-wj6dr\" (UID: \"1f6b477e-a0fd-49c8-896d-c268a6c06d3f\") " pod="openshift-image-registry/node-ca-wj6dr" Nov 04 03:16:44.210875 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.210809 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-run-systemd\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.210875 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.210833 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/923f8598-9df4-46bb-8da0-73442c97522f-agent-certs\") pod \"konnectivity-agent-pmv6n\" (UID: \"923f8598-9df4-46bb-8da0-73442c97522f\") " pod="kube-system/konnectivity-agent-pmv6n" Nov 04 03:16:44.210875 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.210846 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-node-log\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.210875 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:44.210852 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 03:16:44.211446 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:44.210944 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/834b5587-1820-4089-a925-ab992fe654b4-original-pull-secret podName:834b5587-1820-4089-a925-ab992fe654b4 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:44.710906264 +0000 UTC m=+2.308454063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/834b5587-1820-4089-a925-ab992fe654b4-original-pull-secret") pod "global-pull-secret-syncer-8rmjv" (UID: "834b5587-1820-4089-a925-ab992fe654b4") : object "kube-system"/"original-pull-secret" not registered Nov 04 03:16:44.211446 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.210856 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-multus-cni-dir\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:44.211446 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.210988 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q5p48\" (UID: \"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab\") " pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:16:44.211446 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.211027 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-run-ovn\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.211446 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.211053 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-etc-kubernetes\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:44.211446 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.211063 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-multus-cni-dir\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:44.211446 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.211092 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e5e3933f-6610-46a7-934a-8ce205da98d9-sys-fs\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:16:44.211446 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.211104 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-run-ovn\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.211446 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.211128 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-run\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:44.211446 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.211138 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-etc-kubernetes\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:44.211446 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.211157 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngsgw\" (UniqueName: \"kubernetes.io/projected/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-kube-api-access-ngsgw\") pod \"multus-additional-cni-plugins-q5p48\" (UID: \"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab\") " pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:16:44.211446 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.211162 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q5p48\" (UID: \"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab\") " pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:16:44.211446 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.211176 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e5e3933f-6610-46a7-934a-8ce205da98d9-sys-fs\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:16:44.211446 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.211184 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-kubelet\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.211446 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.211204 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-run\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:44.211446 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.211210 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-log-socket\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.211446 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.211238 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-host-var-lib-cni-multus\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:44.212138 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.211259 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-kubelet\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.212138 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.211267 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/834b5587-1820-4089-a925-ab992fe654b4-kubelet-config\") pod \"global-pull-secret-syncer-8rmjv\" (UID: \"834b5587-1820-4089-a925-ab992fe654b4\") " pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:16:44.212138 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.211269 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-cni-binary-copy\") pod \"multus-additional-cni-plugins-q5p48\" (UID: \"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab\") " pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:16:44.212138 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.211297 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/834b5587-1820-4089-a925-ab992fe654b4-dbus\") pod \"global-pull-secret-syncer-8rmjv\" (UID: \"834b5587-1820-4089-a925-ab992fe654b4\") " pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:16:44.212138 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.211327 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-host-var-lib-cni-multus\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:44.212138 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.211319 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-log-socket\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.212138 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.211363 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/834b5587-1820-4089-a925-ab992fe654b4-dbus\") pod \"global-pull-secret-syncer-8rmjv\" (UID: \"834b5587-1820-4089-a925-ab992fe654b4\") " pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:16:44.212138 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.211389 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/834b5587-1820-4089-a925-ab992fe654b4-kubelet-config\") pod \"global-pull-secret-syncer-8rmjv\" (UID: \"834b5587-1820-4089-a925-ab992fe654b4\") " pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:16:44.212138 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.211481 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdtwq\" (UniqueName: \"kubernetes.io/projected/fb15c957-99e8-4c15-b382-788025661809-kube-api-access-cdtwq\") pod \"node-resolver-r9xhm\" (UID: \"fb15c957-99e8-4c15-b382-788025661809\") " pod="openshift-dns/node-resolver-r9xhm" Nov 04 03:16:44.212138 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.211540 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-run-netns\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.212138 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.211564 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-run-openvswitch\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.212138 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.211588 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-host-var-lib-cni-bin\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:44.212138 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.211611 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-lib-modules\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:44.212138 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.211655 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca5887c7-01d6-44ee-ac30-8b4374663d3e-tmp\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:44.212138 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.211691 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fb15c957-99e8-4c15-b382-788025661809-hosts-file\") pod \"node-resolver-r9xhm\" (UID: \"fb15c957-99e8-4c15-b382-788025661809\") " pod="openshift-dns/node-resolver-r9xhm" Nov 04 03:16:44.212138 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.211716 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb15c957-99e8-4c15-b382-788025661809-tmp-dir\") pod \"node-resolver-r9xhm\" (UID: \"fb15c957-99e8-4c15-b382-788025661809\") " pod="openshift-dns/node-resolver-r9xhm" Nov 04 03:16:44.212138 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.211730 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-host-var-lib-cni-bin\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:44.212138 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.211739 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-cni-bin\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.212902 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.211791 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-cni-netd\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.212902 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.211797 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-cni-bin\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.212902 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.211833 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.212902 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.212315 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1f6b477e-a0fd-49c8-896d-c268a6c06d3f-serviceca\") pod \"node-ca-wj6dr\" (UID: \"1f6b477e-a0fd-49c8-896d-c268a6c06d3f\") " pod="openshift-image-registry/node-ca-wj6dr" Nov 04 03:16:44.212902 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.212361 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9f7d1536-9b78-4f13-ae4e-349db78d213e-iptables-alerter-script\") pod \"iptables-alerter-s2zcf\" (UID: \"9f7d1536-9b78-4f13-ae4e-349db78d213e\") " pod="openshift-network-operator/iptables-alerter-s2zcf" Nov 04 03:16:44.212902 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.212399 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-run-ovn-kubernetes\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.212902 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.212442 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcw55\" (UniqueName: \"kubernetes.io/projected/9f7d1536-9b78-4f13-ae4e-349db78d213e-kube-api-access-kcw55\") pod \"iptables-alerter-s2zcf\" (UID: \"9f7d1536-9b78-4f13-ae4e-349db78d213e\") " pod="openshift-network-operator/iptables-alerter-s2zcf" Nov 04 03:16:44.212902 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.212480 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ca254b76-1d53-44aa-81e5-e40a81265145-multus-daemon-config\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:44.212902 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.212533 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e5e3933f-6610-46a7-934a-8ce205da98d9-socket-dir\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:16:44.212902 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.212577 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-modprobe-d\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:44.212902 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.212607 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-var-lib-openvswitch\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.212902 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.212668 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-etc-openvswitch\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.212902 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.212765 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-multus-conf-dir\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:44.212902 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.212800 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q5p48\" (UID: \"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab\") " pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:16:44.212902 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.212830 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-systemd-units\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.213526 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.212912 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-cni-netd\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.213526 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.212974 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.213526 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.213018 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-run-openvswitch\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.213526 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.213184 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-systemd-units\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.213526 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.213225 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-lib-modules\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:44.213526 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.213153 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb15c957-99e8-4c15-b382-788025661809-tmp-dir\") pod \"node-resolver-r9xhm\" (UID: \"fb15c957-99e8-4c15-b382-788025661809\") " pod="openshift-dns/node-resolver-r9xhm" Nov 04 03:16:44.213526 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.213279 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fb15c957-99e8-4c15-b382-788025661809-hosts-file\") pod \"node-resolver-r9xhm\" (UID: \"fb15c957-99e8-4c15-b382-788025661809\") " pod="openshift-dns/node-resolver-r9xhm" Nov 04 03:16:44.213526 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.213354 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-run-netns\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.213526 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.213427 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-etc-openvswitch\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.213526 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.213389 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-run-ovn-kubernetes\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.213526 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.213508 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-modprobe-d\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:44.214063 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.213562 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-var-lib-openvswitch\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.214063 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.213600 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e5e3933f-6610-46a7-934a-8ce205da98d9-socket-dir\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:16:44.214063 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.213950 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-multus-conf-dir\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:44.214288 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.214266 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1f6b477e-a0fd-49c8-896d-c268a6c06d3f-serviceca\") pod \"node-ca-wj6dr\" (UID: \"1f6b477e-a0fd-49c8-896d-c268a6c06d3f\") " pod="openshift-image-registry/node-ca-wj6dr" Nov 04 03:16:44.214546 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.214517 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q5p48\" (UID: \"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab\") " pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:16:44.214641 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.214542 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/923f8598-9df4-46bb-8da0-73442c97522f-agent-certs\") pod \"konnectivity-agent-pmv6n\" (UID: \"923f8598-9df4-46bb-8da0-73442c97522f\") " pod="kube-system/konnectivity-agent-pmv6n" Nov 04 03:16:44.214641 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.214562 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9f7d1536-9b78-4f13-ae4e-349db78d213e-iptables-alerter-script\") pod \"iptables-alerter-s2zcf\" (UID: \"9f7d1536-9b78-4f13-ae4e-349db78d213e\") " pod="openshift-network-operator/iptables-alerter-s2zcf" Nov 04 03:16:44.214743 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.214662 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7xg7\" (UniqueName: \"kubernetes.io/projected/719911b7-0c11-4c79-bb86-fdc043d9f9ab-kube-api-access-n7xg7\") pod \"network-check-target-2hjpn\" (UID: \"719911b7-0c11-4c79-bb86-fdc043d9f9ab\") " pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:16:44.214743 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.214729 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9f7d1536-9b78-4f13-ae4e-349db78d213e-host-slash\") pod \"iptables-alerter-s2zcf\" (UID: \"9f7d1536-9b78-4f13-ae4e-349db78d213e\") " pod="openshift-network-operator/iptables-alerter-s2zcf" Nov 04 03:16:44.214828 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.214755 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e5e3933f-6610-46a7-934a-8ce205da98d9-registration-dir\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:16:44.214828 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.214787 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-sysctl-conf\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:44.214911 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.214833 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aa3f04c-b684-4426-8d46-a36d8c4e77be-metrics-certs\") pod \"network-metrics-daemon-dzg2c\" (UID: \"1aa3f04c-b684-4426-8d46-a36d8c4e77be\") " pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:16:44.214911 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.214884 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/923f8598-9df4-46bb-8da0-73442c97522f-konnectivity-ca\") pod \"konnectivity-agent-pmv6n\" (UID: \"923f8598-9df4-46bb-8da0-73442c97522f\") " pod="kube-system/konnectivity-agent-pmv6n" Nov 04 03:16:44.215002 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.214913 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e5e3933f-6610-46a7-934a-8ce205da98d9-device-dir\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:16:44.215002 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.214944 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ca254b76-1d53-44aa-81e5-e40a81265145-multus-daemon-config\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:44.215002 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.214942 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/326a1269-c427-4470-86a6-70ee348c8791-ovnkube-script-lib\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.215002 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.214987 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdrr8\" (UniqueName: \"kubernetes.io/projected/1aa3f04c-b684-4426-8d46-a36d8c4e77be-kube-api-access-zdrr8\") pod \"network-metrics-daemon-dzg2c\" (UID: \"1aa3f04c-b684-4426-8d46-a36d8c4e77be\") " pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:16:44.215167 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.215012 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e5e3933f-6610-46a7-934a-8ce205da98d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:16:44.215167 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.215053 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-sysctl-conf\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:44.215252 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:44.215227 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 03:16:44.215252 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.215226 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9f7d1536-9b78-4f13-ae4e-349db78d213e-host-slash\") pod \"iptables-alerter-s2zcf\" (UID: \"9f7d1536-9b78-4f13-ae4e-349db78d213e\") " pod="openshift-network-operator/iptables-alerter-s2zcf" Nov 04 03:16:44.215338 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.215306 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e5e3933f-6610-46a7-934a-8ce205da98d9-registration-dir\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:16:44.215338 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:44.215317 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aa3f04c-b684-4426-8d46-a36d8c4e77be-metrics-certs podName:1aa3f04c-b684-4426-8d46-a36d8c4e77be nodeName:}" failed. No retries permitted until 2026-11-04 03:16:44.715299391 +0000 UTC m=+2.312847181 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1aa3f04c-b684-4426-8d46-a36d8c4e77be-metrics-certs") pod "network-metrics-daemon-dzg2c" (UID: "1aa3f04c-b684-4426-8d46-a36d8c4e77be") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 03:16:44.215338 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.215054 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q5p48\" (UID: \"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab\") " pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:16:44.215477 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.215355 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca254b76-1d53-44aa-81e5-e40a81265145-cni-binary-copy\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:44.215477 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.215379 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-sysconfig\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:44.215477 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.215408 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-kubernetes\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:44.215477 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.215436 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-sysctl-d\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:44.215477 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.215462 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-host\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:44.215709 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.215490 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f6b477e-a0fd-49c8-896d-c268a6c06d3f-host\") pod \"node-ca-wj6dr\" (UID: \"1f6b477e-a0fd-49c8-896d-c268a6c06d3f\") " pod="openshift-image-registry/node-ca-wj6dr" Nov 04 03:16:44.215709 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.215514 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-host-run-k8s-cni-cncf-io\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:44.215709 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.215541 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-host-run-netns\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:44.215709 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.215600 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmxc6\" (UniqueName: \"kubernetes.io/projected/e5e3933f-6610-46a7-934a-8ce205da98d9-kube-api-access-zmxc6\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:16:44.215709 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.215675 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q24vc\" (UniqueName: \"kubernetes.io/projected/ca5887c7-01d6-44ee-ac30-8b4374663d3e-kube-api-access-q24vc\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:44.215709 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.215699 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/326a1269-c427-4470-86a6-70ee348c8791-ovnkube-config\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.215967 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.215753 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-host-var-lib-kubelet\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:44.215967 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.215892 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-host-var-lib-kubelet\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:44.216057 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.215946 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/923f8598-9df4-46bb-8da0-73442c97522f-konnectivity-ca\") pod \"konnectivity-agent-pmv6n\" (UID: \"923f8598-9df4-46bb-8da0-73442c97522f\") " pod="kube-system/konnectivity-agent-pmv6n" Nov 04 03:16:44.216057 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.216039 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e5e3933f-6610-46a7-934a-8ce205da98d9-device-dir\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:16:44.216470 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.216450 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca254b76-1d53-44aa-81e5-e40a81265145-cni-binary-copy\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:44.216549 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.216514 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-sysconfig\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:44.216612 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.216571 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-kubernetes\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:44.216735 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.216719 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-sysctl-d\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:44.216782 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.216762 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-host\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:44.216829 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.216807 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f6b477e-a0fd-49c8-896d-c268a6c06d3f-host\") pod \"node-ca-wj6dr\" (UID: \"1f6b477e-a0fd-49c8-896d-c268a6c06d3f\") " pod="openshift-image-registry/node-ca-wj6dr" Nov 04 03:16:44.216938 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.216901 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-host-run-k8s-cni-cncf-io\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:44.216987 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.216923 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e5e3933f-6610-46a7-934a-8ce205da98d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:16:44.217031 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.216978 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-host-run-netns\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:44.217828 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.217807 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-tuned\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:44.217935 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.217919 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/326a1269-c427-4470-86a6-70ee348c8791-ovnkube-script-lib\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.217991 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.217929 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/326a1269-c427-4470-86a6-70ee348c8791-ovnkube-config\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.217991 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.217918 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q5p48\" (UID: \"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab\") " pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:16:44.218198 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.218166 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/326a1269-c427-4470-86a6-70ee348c8791-ovn-node-metrics-cert\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.219740 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.219718 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca5887c7-01d6-44ee-ac30-8b4374663d3e-tmp\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:44.271502 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.271429 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjwjp\" (UniqueName: \"kubernetes.io/projected/1f6b477e-a0fd-49c8-896d-c268a6c06d3f-kube-api-access-vjwjp\") pod \"node-ca-wj6dr\" (UID: \"1f6b477e-a0fd-49c8-896d-c268a6c06d3f\") " pod="openshift-image-registry/node-ca-wj6dr" Nov 04 03:16:44.271673 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.271563 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qzd8\" (UniqueName: \"kubernetes.io/projected/326a1269-c427-4470-86a6-70ee348c8791-kube-api-access-8qzd8\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.271673 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.271601 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngsgw\" (UniqueName: \"kubernetes.io/projected/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-kube-api-access-ngsgw\") pod \"multus-additional-cni-plugins-q5p48\" (UID: \"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab\") " pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:16:44.272695 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.272668 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqc8z\" (UniqueName: \"kubernetes.io/projected/ca254b76-1d53-44aa-81e5-e40a81265145-kube-api-access-fqc8z\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:16:44.272695 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.272694 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdtwq\" (UniqueName: \"kubernetes.io/projected/fb15c957-99e8-4c15-b382-788025661809-kube-api-access-cdtwq\") pod \"node-resolver-r9xhm\" (UID: \"fb15c957-99e8-4c15-b382-788025661809\") " pod="openshift-dns/node-resolver-r9xhm" Nov 04 03:16:44.276121 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:44.276095 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 03:16:44.276282 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:44.276128 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 03:16:44.276282 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:44.276145 2578 projected.go:196] Error preparing data for projected volume kube-api-access-n7xg7 for pod openshift-network-diagnostics/network-check-target-2hjpn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 03:16:44.276282 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:44.276230 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/719911b7-0c11-4c79-bb86-fdc043d9f9ab-kube-api-access-n7xg7 podName:719911b7-0c11-4c79-bb86-fdc043d9f9ab nodeName:}" failed. No retries permitted until 2026-11-04 03:16:44.776208245 +0000 UTC m=+2.373756040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-n7xg7" (UniqueName: "kubernetes.io/projected/719911b7-0c11-4c79-bb86-fdc043d9f9ab-kube-api-access-n7xg7") pod "network-check-target-2hjpn" (UID: "719911b7-0c11-4c79-bb86-fdc043d9f9ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 03:16:44.277163 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.277142 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcw55\" (UniqueName: \"kubernetes.io/projected/9f7d1536-9b78-4f13-ae4e-349db78d213e-kube-api-access-kcw55\") pod \"iptables-alerter-s2zcf\" (UID: \"9f7d1536-9b78-4f13-ae4e-349db78d213e\") " pod="openshift-network-operator/iptables-alerter-s2zcf" Nov 04 03:16:44.278109 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.278085 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdrr8\" (UniqueName: \"kubernetes.io/projected/1aa3f04c-b684-4426-8d46-a36d8c4e77be-kube-api-access-zdrr8\") pod \"network-metrics-daemon-dzg2c\" (UID: \"1aa3f04c-b684-4426-8d46-a36d8c4e77be\") " pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:16:44.278266 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.278252 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q24vc\" (UniqueName: \"kubernetes.io/projected/ca5887c7-01d6-44ee-ac30-8b4374663d3e-kube-api-access-q24vc\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:44.282272 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.282250 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmxc6\" (UniqueName: \"kubernetes.io/projected/e5e3933f-6610-46a7-934a-8ce205da98d9-kube-api-access-zmxc6\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:16:44.377341 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.377298 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r9xhm" Nov 04 03:16:44.383433 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.383412 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wj6dr" Nov 04 03:16:44.387098 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.387075 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t2twg" Nov 04 03:16:44.423668 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.423634 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:16:44.427377 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.427358 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:16:44.433060 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.433040 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:16:44.433790 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.433775 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 03:16:44.456416 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.456395 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pmv6n" Nov 04 03:16:44.466640 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:44.466598 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca5887c7_01d6_44ee_ac30_8b4374663d3e.slice/crio-94f78e81b681ee7be5000043942699b08bae5f9c2a5d57070f9b1a59a7701ca0 WatchSource:0}: Error finding container 94f78e81b681ee7be5000043942699b08bae5f9c2a5d57070f9b1a59a7701ca0: Status 404 returned error can't find the container with id 94f78e81b681ee7be5000043942699b08bae5f9c2a5d57070f9b1a59a7701ca0 Nov 04 03:16:44.467257 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:44.467237 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12436f4d0565699f03041c74b80c999a.slice/crio-d334d5ffb746244dc6e3456097b41613e54e96dfd4dbbb4e77995ab0fd416185 WatchSource:0}: Error finding container d334d5ffb746244dc6e3456097b41613e54e96dfd4dbbb4e77995ab0fd416185: Status 404 returned error can't find the container with id d334d5ffb746244dc6e3456097b41613e54e96dfd4dbbb4e77995ab0fd416185 Nov 04 03:16:44.467929 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:44.467907 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca990c51a9c4be7e8fb143b232789941.slice/crio-79ce3567247d25712b2d4037523f0437b4da86f3d90d0bac8859b00d5cf3eeee WatchSource:0}: Error finding container 79ce3567247d25712b2d4037523f0437b4da86f3d90d0bac8859b00d5cf3eeee: Status 404 returned error can't find the container with id 79ce3567247d25712b2d4037523f0437b4da86f3d90d0bac8859b00d5cf3eeee Nov 04 03:16:44.470409 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:44.470387 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod923f8598_9df4_46bb_8da0_73442c97522f.slice/crio-988dc3031df7bc31a97c6a25b8e24cda782cef441f820edcda4057223ffb7655 WatchSource:0}: Error finding container 988dc3031df7bc31a97c6a25b8e24cda782cef441f820edcda4057223ffb7655: Status 404 returned error can't find the container with id 988dc3031df7bc31a97c6a25b8e24cda782cef441f820edcda4057223ffb7655 Nov 04 03:16:44.472942 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.472929 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 03:16:44.482698 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.482678 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s2zcf" Nov 04 03:16:44.490309 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.490289 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:16:44.492003 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:44.491969 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f7d1536_9b78_4f13_ae4e_349db78d213e.slice/crio-6ec7ae50a4a7d2295b6890493085aaf6410dcfdc92d906ba9d10043571c9812b WatchSource:0}: Error finding container 6ec7ae50a4a7d2295b6890493085aaf6410dcfdc92d906ba9d10043571c9812b: Status 404 returned error can't find the container with id 6ec7ae50a4a7d2295b6890493085aaf6410dcfdc92d906ba9d10043571c9812b Nov 04 03:16:44.498463 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:44.498440 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96ce4e9b_b61b_4769_bbc9_7e0b4174d2ab.slice/crio-93b3cbc9035ec7cec6ac37617c10933127bb5177af65587c6898c97cb381c6cd WatchSource:0}: Error finding container 93b3cbc9035ec7cec6ac37617c10933127bb5177af65587c6898c97cb381c6cd: Status 404 returned error can't find the container with id 93b3cbc9035ec7cec6ac37617c10933127bb5177af65587c6898c97cb381c6cd Nov 04 03:16:44.719165 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.719069 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aa3f04c-b684-4426-8d46-a36d8c4e77be-metrics-certs\") pod \"network-metrics-daemon-dzg2c\" (UID: \"1aa3f04c-b684-4426-8d46-a36d8c4e77be\") " pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:16:44.719165 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.719146 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/834b5587-1820-4089-a925-ab992fe654b4-original-pull-secret\") pod \"global-pull-secret-syncer-8rmjv\" (UID: \"834b5587-1820-4089-a925-ab992fe654b4\") " pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:16:44.719362 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:44.719231 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 03:16:44.719362 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:44.719270 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 03:16:44.719362 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:44.719304 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aa3f04c-b684-4426-8d46-a36d8c4e77be-metrics-certs podName:1aa3f04c-b684-4426-8d46-a36d8c4e77be nodeName:}" failed. No retries permitted until 2026-11-04 03:16:45.719289232 +0000 UTC m=+3.316837004 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1aa3f04c-b684-4426-8d46-a36d8c4e77be-metrics-certs") pod "network-metrics-daemon-dzg2c" (UID: "1aa3f04c-b684-4426-8d46-a36d8c4e77be") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 03:16:44.719362 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:44.719325 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/834b5587-1820-4089-a925-ab992fe654b4-original-pull-secret podName:834b5587-1820-4089-a925-ab992fe654b4 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:45.719310654 +0000 UTC m=+3.316858426 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/834b5587-1820-4089-a925-ab992fe654b4-original-pull-secret") pod "global-pull-secret-syncer-8rmjv" (UID: "834b5587-1820-4089-a925-ab992fe654b4") : object "kube-system"/"original-pull-secret" not registered Nov 04 03:16:44.820341 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.820300 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7xg7\" (UniqueName: \"kubernetes.io/projected/719911b7-0c11-4c79-bb86-fdc043d9f9ab-kube-api-access-n7xg7\") pod \"network-check-target-2hjpn\" (UID: \"719911b7-0c11-4c79-bb86-fdc043d9f9ab\") " pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:16:44.820980 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:44.820455 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 03:16:44.820980 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:44.820474 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 03:16:44.820980 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:44.820483 2578 projected.go:196] Error preparing data for projected volume kube-api-access-n7xg7 for pod openshift-network-diagnostics/network-check-target-2hjpn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 03:16:44.820980 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:44.820535 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/719911b7-0c11-4c79-bb86-fdc043d9f9ab-kube-api-access-n7xg7 podName:719911b7-0c11-4c79-bb86-fdc043d9f9ab nodeName:}" failed. No retries permitted until 2026-11-04 03:16:45.820520111 +0000 UTC m=+3.418067896 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-n7xg7" (UniqueName: "kubernetes.io/projected/719911b7-0c11-4c79-bb86-fdc043d9f9ab-kube-api-access-n7xg7") pod "network-check-target-2hjpn" (UID: "719911b7-0c11-4c79-bb86-fdc043d9f9ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 03:16:44.933676 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.933607 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pmv6n" event={"ID":"923f8598-9df4-46bb-8da0-73442c97522f","Type":"ContainerStarted","Data":"988dc3031df7bc31a97c6a25b8e24cda782cef441f820edcda4057223ffb7655"} Nov 04 03:16:44.934678 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.934653 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-6.us-west-1.compute.internal" event={"ID":"ca990c51a9c4be7e8fb143b232789941","Type":"ContainerStarted","Data":"79ce3567247d25712b2d4037523f0437b4da86f3d90d0bac8859b00d5cf3eeee"} Nov 04 03:16:44.936131 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.936104 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal" event={"ID":"12436f4d0565699f03041c74b80c999a","Type":"ContainerStarted","Data":"d334d5ffb746244dc6e3456097b41613e54e96dfd4dbbb4e77995ab0fd416185"} Nov 04 03:16:44.937158 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.937139 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" event={"ID":"ca5887c7-01d6-44ee-ac30-8b4374663d3e","Type":"ContainerStarted","Data":"94f78e81b681ee7be5000043942699b08bae5f9c2a5d57070f9b1a59a7701ca0"} Nov 04 03:16:44.938219 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.938201 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q5p48" event={"ID":"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab","Type":"ContainerStarted","Data":"93b3cbc9035ec7cec6ac37617c10933127bb5177af65587c6898c97cb381c6cd"} Nov 04 03:16:44.939192 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:44.939175 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s2zcf" event={"ID":"9f7d1536-9b78-4f13-ae4e-349db78d213e","Type":"ContainerStarted","Data":"6ec7ae50a4a7d2295b6890493085aaf6410dcfdc92d906ba9d10043571c9812b"} Nov 04 03:16:45.048210 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:45.048173 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 03:16:45.196413 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:45.196382 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f6b477e_a0fd_49c8_896d_c268a6c06d3f.slice/crio-bd778358e255e28aa0c96fbbfcca8600ba0832bf0e6a2c63ac455194cecfc0a8 WatchSource:0}: Error finding container bd778358e255e28aa0c96fbbfcca8600ba0832bf0e6a2c63ac455194cecfc0a8: Status 404 returned error can't find the container with id bd778358e255e28aa0c96fbbfcca8600ba0832bf0e6a2c63ac455194cecfc0a8 Nov 04 03:16:45.197769 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:45.197728 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 03:11:43 +0000 UTC" deadline="2028-04-19 11:33:56.64836554 +0000 UTC" Nov 04 03:16:45.197840 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:45.197771 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12776h17m11.450598367s" Nov 04 03:16:45.210017 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:45.209993 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 03:16:45.297136 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:45.297064 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod326a1269_c427_4470_86a6_70ee348c8791.slice/crio-ba74b6466c9db3a2e9b4811b48d639a47993d10088c68fda03f7053e5d099e10 WatchSource:0}: Error finding container ba74b6466c9db3a2e9b4811b48d639a47993d10088c68fda03f7053e5d099e10: Status 404 returned error can't find the container with id ba74b6466c9db3a2e9b4811b48d639a47993d10088c68fda03f7053e5d099e10 Nov 04 03:16:45.307004 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:45.306976 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca254b76_1d53_44aa_81e5_e40a81265145.slice/crio-c777436c764c665c4bb019ddad6db5c86ce03725b182ddcb23c951590f8595af WatchSource:0}: Error finding container c777436c764c665c4bb019ddad6db5c86ce03725b182ddcb23c951590f8595af: Status 404 returned error can't find the container with id c777436c764c665c4bb019ddad6db5c86ce03725b182ddcb23c951590f8595af Nov 04 03:16:45.328900 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:45.328869 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb15c957_99e8_4c15_b382_788025661809.slice/crio-64803c7435bf8b87b711ba3436cb3778c5f7bebc554c6d345f97132b5a0c8104 WatchSource:0}: Error finding container 64803c7435bf8b87b711ba3436cb3778c5f7bebc554c6d345f97132b5a0c8104: Status 404 returned error can't find the container with id 64803c7435bf8b87b711ba3436cb3778c5f7bebc554c6d345f97132b5a0c8104 Nov 04 03:16:45.349554 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:16:45.349525 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e3933f_6610_46a7_934a_8ce205da98d9.slice/crio-1846dbcff1a501528f398dfde9d918e157bceba20e37ce363fa71726ac064d1e WatchSource:0}: Error finding container 1846dbcff1a501528f398dfde9d918e157bceba20e37ce363fa71726ac064d1e: Status 404 returned error can't find the container with id 1846dbcff1a501528f398dfde9d918e157bceba20e37ce363fa71726ac064d1e Nov 04 03:16:45.727644 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:45.727176 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/834b5587-1820-4089-a925-ab992fe654b4-original-pull-secret\") pod \"global-pull-secret-syncer-8rmjv\" (UID: \"834b5587-1820-4089-a925-ab992fe654b4\") " pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:16:45.727644 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:45.727277 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aa3f04c-b684-4426-8d46-a36d8c4e77be-metrics-certs\") pod \"network-metrics-daemon-dzg2c\" (UID: \"1aa3f04c-b684-4426-8d46-a36d8c4e77be\") " pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:16:45.727644 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:45.727426 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 03:16:45.727644 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:45.727489 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aa3f04c-b684-4426-8d46-a36d8c4e77be-metrics-certs podName:1aa3f04c-b684-4426-8d46-a36d8c4e77be nodeName:}" failed. No retries permitted until 2026-11-04 03:16:47.727471296 +0000 UTC m=+5.325019087 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1aa3f04c-b684-4426-8d46-a36d8c4e77be-metrics-certs") pod "network-metrics-daemon-dzg2c" (UID: "1aa3f04c-b684-4426-8d46-a36d8c4e77be") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 03:16:45.727644 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:45.727553 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 03:16:45.727644 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:45.727586 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/834b5587-1820-4089-a925-ab992fe654b4-original-pull-secret podName:834b5587-1820-4089-a925-ab992fe654b4 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:47.727575016 +0000 UTC m=+5.325122789 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/834b5587-1820-4089-a925-ab992fe654b4-original-pull-secret") pod "global-pull-secret-syncer-8rmjv" (UID: "834b5587-1820-4089-a925-ab992fe654b4") : object "kube-system"/"original-pull-secret" not registered Nov 04 03:16:45.828104 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:45.828052 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7xg7\" (UniqueName: \"kubernetes.io/projected/719911b7-0c11-4c79-bb86-fdc043d9f9ab-kube-api-access-n7xg7\") pod \"network-check-target-2hjpn\" (UID: \"719911b7-0c11-4c79-bb86-fdc043d9f9ab\") " pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:16:45.828772 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:45.828266 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 03:16:45.828772 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:45.828284 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 03:16:45.828772 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:45.828296 2578 projected.go:196] Error preparing data for projected volume kube-api-access-n7xg7 for pod openshift-network-diagnostics/network-check-target-2hjpn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 03:16:45.828772 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:45.828356 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/719911b7-0c11-4c79-bb86-fdc043d9f9ab-kube-api-access-n7xg7 podName:719911b7-0c11-4c79-bb86-fdc043d9f9ab nodeName:}" failed. No retries permitted until 2026-11-04 03:16:47.828337945 +0000 UTC m=+5.425885731 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-n7xg7" (UniqueName: "kubernetes.io/projected/719911b7-0c11-4c79-bb86-fdc043d9f9ab-kube-api-access-n7xg7") pod "network-check-target-2hjpn" (UID: "719911b7-0c11-4c79-bb86-fdc043d9f9ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 03:16:45.930158 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:45.930065 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:16:45.930365 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:45.930202 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hjpn" podUID="719911b7-0c11-4c79-bb86-fdc043d9f9ab" Nov 04 03:16:45.930365 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:45.930290 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:16:45.930463 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:45.930361 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rmjv" podUID="834b5587-1820-4089-a925-ab992fe654b4" Nov 04 03:16:45.930463 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:45.930424 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:16:45.930564 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:45.930496 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzg2c" podUID="1aa3f04c-b684-4426-8d46-a36d8c4e77be" Nov 04 03:16:45.943744 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:45.943705 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" event={"ID":"e5e3933f-6610-46a7-934a-8ce205da98d9","Type":"ContainerStarted","Data":"1846dbcff1a501528f398dfde9d918e157bceba20e37ce363fa71726ac064d1e"} Nov 04 03:16:45.948676 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:45.948639 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r9xhm" event={"ID":"fb15c957-99e8-4c15-b382-788025661809","Type":"ContainerStarted","Data":"64803c7435bf8b87b711ba3436cb3778c5f7bebc554c6d345f97132b5a0c8104"} Nov 04 03:16:45.950274 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:45.950244 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wj6dr" event={"ID":"1f6b477e-a0fd-49c8-896d-c268a6c06d3f","Type":"ContainerStarted","Data":"bd778358e255e28aa0c96fbbfcca8600ba0832bf0e6a2c63ac455194cecfc0a8"} Nov 04 03:16:45.951522 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:45.951496 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t2twg" event={"ID":"ca254b76-1d53-44aa-81e5-e40a81265145","Type":"ContainerStarted","Data":"c777436c764c665c4bb019ddad6db5c86ce03725b182ddcb23c951590f8595af"} Nov 04 03:16:45.953988 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:45.953963 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" event={"ID":"326a1269-c427-4470-86a6-70ee348c8791","Type":"ContainerStarted","Data":"ba74b6466c9db3a2e9b4811b48d639a47993d10088c68fda03f7053e5d099e10"} Nov 04 03:16:46.198708 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:46.198579 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 03:11:43 +0000 UTC" deadline="2028-07-29 20:01:34.815618742 +0000 UTC" Nov 04 03:16:46.198708 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:46.198661 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15208h44m48.616961989s" Nov 04 03:16:47.747651 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:47.747135 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/834b5587-1820-4089-a925-ab992fe654b4-original-pull-secret\") pod \"global-pull-secret-syncer-8rmjv\" (UID: \"834b5587-1820-4089-a925-ab992fe654b4\") " pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:16:47.747651 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:47.747431 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 03:16:47.747651 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:47.747484 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aa3f04c-b684-4426-8d46-a36d8c4e77be-metrics-certs\") pod \"network-metrics-daemon-dzg2c\" (UID: \"1aa3f04c-b684-4426-8d46-a36d8c4e77be\") " pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:16:47.747651 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:47.747518 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/834b5587-1820-4089-a925-ab992fe654b4-original-pull-secret podName:834b5587-1820-4089-a925-ab992fe654b4 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:51.74749538 +0000 UTC m=+9.345043176 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/834b5587-1820-4089-a925-ab992fe654b4-original-pull-secret") pod "global-pull-secret-syncer-8rmjv" (UID: "834b5587-1820-4089-a925-ab992fe654b4") : object "kube-system"/"original-pull-secret" not registered Nov 04 03:16:47.747651 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:47.747581 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 03:16:47.747651 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:47.747649 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aa3f04c-b684-4426-8d46-a36d8c4e77be-metrics-certs podName:1aa3f04c-b684-4426-8d46-a36d8c4e77be nodeName:}" failed. No retries permitted until 2026-11-04 03:16:51.747632045 +0000 UTC m=+9.345179830 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1aa3f04c-b684-4426-8d46-a36d8c4e77be-metrics-certs") pod "network-metrics-daemon-dzg2c" (UID: "1aa3f04c-b684-4426-8d46-a36d8c4e77be") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 03:16:47.849064 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:47.848697 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7xg7\" (UniqueName: \"kubernetes.io/projected/719911b7-0c11-4c79-bb86-fdc043d9f9ab-kube-api-access-n7xg7\") pod \"network-check-target-2hjpn\" (UID: \"719911b7-0c11-4c79-bb86-fdc043d9f9ab\") " pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:16:47.849064 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:47.848886 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 03:16:47.849064 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:47.848907 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 03:16:47.849064 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:47.848919 2578 projected.go:196] Error preparing data for projected volume kube-api-access-n7xg7 for pod openshift-network-diagnostics/network-check-target-2hjpn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 03:16:47.849064 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:47.848979 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/719911b7-0c11-4c79-bb86-fdc043d9f9ab-kube-api-access-n7xg7 podName:719911b7-0c11-4c79-bb86-fdc043d9f9ab nodeName:}" failed. No retries permitted until 2026-11-04 03:16:51.84896237 +0000 UTC m=+9.446510157 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-n7xg7" (UniqueName: "kubernetes.io/projected/719911b7-0c11-4c79-bb86-fdc043d9f9ab-kube-api-access-n7xg7") pod "network-check-target-2hjpn" (UID: "719911b7-0c11-4c79-bb86-fdc043d9f9ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 03:16:47.929992 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:47.929951 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:16:47.930174 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:47.930084 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hjpn" podUID="719911b7-0c11-4c79-bb86-fdc043d9f9ab" Nov 04 03:16:47.930174 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:47.930151 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:16:47.930297 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:47.930220 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rmjv" podUID="834b5587-1820-4089-a925-ab992fe654b4" Nov 04 03:16:47.930297 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:47.930269 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:16:47.930397 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:47.930353 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzg2c" podUID="1aa3f04c-b684-4426-8d46-a36d8c4e77be" Nov 04 03:16:47.970120 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:47.970079 2578 generic.go:356] "Generic (PLEG): container finished" podID="96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab" containerID="06ba6bb8082f6aac9dd2851112ce7b3220b92fb2c0dc5f7cceef6a8dc5cf81e1" exitCode=0 Nov 04 03:16:47.970308 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:47.970194 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q5p48" event={"ID":"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab","Type":"ContainerDied","Data":"06ba6bb8082f6aac9dd2851112ce7b3220b92fb2c0dc5f7cceef6a8dc5cf81e1"} Nov 04 03:16:48.976564 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:48.976472 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pmv6n" event={"ID":"923f8598-9df4-46bb-8da0-73442c97522f","Type":"ContainerStarted","Data":"bd5b2b493b5be0a817749eda9724de7bb6fb0c8ffb3c4c44444f1f0e903a273b"} Nov 04 03:16:49.930190 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:49.930155 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:16:49.930382 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:49.930257 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hjpn" podUID="719911b7-0c11-4c79-bb86-fdc043d9f9ab" Nov 04 03:16:49.930382 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:49.930314 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:16:49.930382 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:49.930361 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rmjv" podUID="834b5587-1820-4089-a925-ab992fe654b4" Nov 04 03:16:49.930569 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:49.930397 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:16:49.930569 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:49.930444 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzg2c" podUID="1aa3f04c-b684-4426-8d46-a36d8c4e77be" Nov 04 03:16:49.992049 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:49.992007 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-6.us-west-1.compute.internal" event={"ID":"ca990c51a9c4be7e8fb143b232789941","Type":"ContainerStarted","Data":"c20a8c1167238ad5e0d857c02273fad6f50a3bab6739b88d986191be44efc5e7"} Nov 04 03:16:50.004339 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:50.004299 2578 generic.go:356] "Generic (PLEG): container finished" podID="12436f4d0565699f03041c74b80c999a" containerID="9198cc3bcfb59628049f7b350ac2652eda5b2d8873955670a914423b2d0f33be" exitCode=0 Nov 04 03:16:50.005334 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:50.005299 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal" event={"ID":"12436f4d0565699f03041c74b80c999a","Type":"ContainerDied","Data":"9198cc3bcfb59628049f7b350ac2652eda5b2d8873955670a914423b2d0f33be"} Nov 04 03:16:50.106131 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:50.106067 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-6.us-west-1.compute.internal" podStartSLOduration=7.106046518 podStartE2EDuration="7.106046518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 03:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:16:50.105688352 +0000 UTC m=+7.703236181" watchObservedRunningTime="2026-11-04 03:16:50.106046518 +0000 UTC m=+7.703594313" Nov 04 03:16:50.106524 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:50.106491 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pmv6n" podStartSLOduration=3.038829027 podStartE2EDuration="7.1064848s" totalImagesPullingTime="4.067655773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:16:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:16:49.094603986 +0000 UTC m=+6.692151780" watchObservedRunningTime="2026-11-04 03:16:50.1064848 +0000 UTC m=+7.704032595" Nov 04 03:16:51.014452 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:51.014417 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal" event={"ID":"12436f4d0565699f03041c74b80c999a","Type":"ContainerStarted","Data":"bcd00165b2b4556e7338124d93b06f75dfd4a739ed25aaa4a8fa96d029a03393"} Nov 04 03:16:51.133121 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:51.133044 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal" podStartSLOduration=8.133021134 podStartE2EDuration="8.133021134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 03:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:16:51.133005561 +0000 UTC m=+8.730553373" watchObservedRunningTime="2026-11-04 03:16:51.133021134 +0000 UTC m=+8.730568932" Nov 04 03:16:51.790967 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:51.790614 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aa3f04c-b684-4426-8d46-a36d8c4e77be-metrics-certs\") pod \"network-metrics-daemon-dzg2c\" (UID: \"1aa3f04c-b684-4426-8d46-a36d8c4e77be\") " pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:16:51.790967 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:51.790702 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/834b5587-1820-4089-a925-ab992fe654b4-original-pull-secret\") pod \"global-pull-secret-syncer-8rmjv\" (UID: \"834b5587-1820-4089-a925-ab992fe654b4\") " pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:16:51.790967 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:51.790765 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 03:16:51.790967 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:51.790778 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 03:16:51.790967 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:51.790830 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/834b5587-1820-4089-a925-ab992fe654b4-original-pull-secret podName:834b5587-1820-4089-a925-ab992fe654b4 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:59.790812632 +0000 UTC m=+17.388360409 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/834b5587-1820-4089-a925-ab992fe654b4-original-pull-secret") pod "global-pull-secret-syncer-8rmjv" (UID: "834b5587-1820-4089-a925-ab992fe654b4") : object "kube-system"/"original-pull-secret" not registered Nov 04 03:16:51.790967 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:51.790849 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aa3f04c-b684-4426-8d46-a36d8c4e77be-metrics-certs podName:1aa3f04c-b684-4426-8d46-a36d8c4e77be nodeName:}" failed. No retries permitted until 2026-11-04 03:16:59.790839155 +0000 UTC m=+17.388386940 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1aa3f04c-b684-4426-8d46-a36d8c4e77be-metrics-certs") pod "network-metrics-daemon-dzg2c" (UID: "1aa3f04c-b684-4426-8d46-a36d8c4e77be") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 03:16:51.891389 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:51.891031 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7xg7\" (UniqueName: \"kubernetes.io/projected/719911b7-0c11-4c79-bb86-fdc043d9f9ab-kube-api-access-n7xg7\") pod \"network-check-target-2hjpn\" (UID: \"719911b7-0c11-4c79-bb86-fdc043d9f9ab\") " pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:16:51.891389 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:51.891210 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 03:16:51.891389 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:51.891236 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 03:16:51.891389 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:51.891250 2578 projected.go:196] Error preparing data for projected volume kube-api-access-n7xg7 for pod openshift-network-diagnostics/network-check-target-2hjpn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 03:16:51.891389 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:51.891322 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/719911b7-0c11-4c79-bb86-fdc043d9f9ab-kube-api-access-n7xg7 podName:719911b7-0c11-4c79-bb86-fdc043d9f9ab nodeName:}" failed. No retries permitted until 2026-11-04 03:16:59.891304095 +0000 UTC m=+17.488851867 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-n7xg7" (UniqueName: "kubernetes.io/projected/719911b7-0c11-4c79-bb86-fdc043d9f9ab-kube-api-access-n7xg7") pod "network-check-target-2hjpn" (UID: "719911b7-0c11-4c79-bb86-fdc043d9f9ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 03:16:51.930700 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:51.930279 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:16:51.930700 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:51.930334 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:16:51.930700 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:51.930424 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hjpn" podUID="719911b7-0c11-4c79-bb86-fdc043d9f9ab" Nov 04 03:16:51.930700 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:51.930471 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:16:51.930700 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:51.930566 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzg2c" podUID="1aa3f04c-b684-4426-8d46-a36d8c4e77be" Nov 04 03:16:51.930700 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:51.930658 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rmjv" podUID="834b5587-1820-4089-a925-ab992fe654b4" Nov 04 03:16:52.023092 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:52.022995 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" event={"ID":"e5e3933f-6610-46a7-934a-8ce205da98d9","Type":"ContainerStarted","Data":"c7673325e7b6d9a64a15caa038630380457f65bef428a392d1fbca8ec6a87c9c"} Nov 04 03:16:52.027754 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:52.027717 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wj6dr" event={"ID":"1f6b477e-a0fd-49c8-896d-c268a6c06d3f","Type":"ContainerStarted","Data":"4a9ff8207a4561935badbfb30d83ca10b6c60a4d6d1cfebc26bbe3540d2cce21"} Nov 04 03:16:53.128051 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:53.127380 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pmv6n" Nov 04 03:16:53.129338 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:53.129112 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pmv6n" Nov 04 03:16:53.245160 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:53.244148 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wj6dr" podStartSLOduration=4.469513388 podStartE2EDuration="10.244126122s" totalImagesPullingTime="5.774612734s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 03:16:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:16:52.142255106 +0000 UTC m=+9.739802917" watchObservedRunningTime="2026-11-04 03:16:53.244126122 +0000 UTC m=+10.841673954" Nov 04 03:16:53.921667 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:53.921605 2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 04 03:16:53.930430 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:53.929960 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:16:53.930430 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:53.930105 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzg2c" podUID="1aa3f04c-b684-4426-8d46-a36d8c4e77be" Nov 04 03:16:53.930430 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:53.930172 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:16:53.930430 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:53.930255 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hjpn" podUID="719911b7-0c11-4c79-bb86-fdc043d9f9ab" Nov 04 03:16:53.930430 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:53.930297 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:16:53.930430 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:53.930360 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rmjv" podUID="834b5587-1820-4089-a925-ab992fe654b4" Nov 04 03:16:54.037530 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:54.037497 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r9xhm" event={"ID":"fb15c957-99e8-4c15-b382-788025661809","Type":"ContainerStarted","Data":"649ff569ea72360ffd61a59509694f1fed21eabedab4a6eb8c2d53051204d0ae"} Nov 04 03:16:54.041404 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:54.041365 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t2twg" event={"ID":"ca254b76-1d53-44aa-81e5-e40a81265145","Type":"ContainerStarted","Data":"931aa1d8edeac865d8e459304b314733d82b5051783da0851bb48d75c96b0f39"} Nov 04 03:16:54.043732 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:54.043700 2578 generic.go:356] "Generic (PLEG): container finished" podID="96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab" containerID="6721ba231dc26a949079274cd0723294ef0a0eac51544cb55964e4bd6aeae9d1" exitCode=0 Nov 04 03:16:54.043950 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:54.043773 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q5p48" event={"ID":"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab","Type":"ContainerDied","Data":"6721ba231dc26a949079274cd0723294ef0a0eac51544cb55964e4bd6aeae9d1"} Nov 04 03:16:54.047412 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:54.047383 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" event={"ID":"e5e3933f-6610-46a7-934a-8ce205da98d9","Type":"ContainerStarted","Data":"b17dceda5c1f41772d808f501502075cb41c24955edc23c6fbdb89942fe7676c"} Nov 04 03:16:54.157113 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:54.156990 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-r9xhm" podStartSLOduration=2.630172907 podStartE2EDuration="11.156970841s" totalImagesPullingTime="8.526797934s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 03:16:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:16:54.156196709 +0000 UTC m=+11.753744519" watchObservedRunningTime="2026-11-04 03:16:54.156970841 +0000 UTC m=+11.754518637" Nov 04 03:16:54.877658 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:54.876897 2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-04T03:16:53.921641784Z","UUID":"26e594da-50e5-4060-8fe2-faf5ca34e582","Handler":null,"Name":"","Endpoint":""} Nov 04 03:16:54.879583 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:54.879560 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 04 03:16:54.879583 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:54.879589 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 04 03:16:55.058122 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:55.058084 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s2zcf" event={"ID":"9f7d1536-9b78-4f13-ae4e-349db78d213e","Type":"ContainerStarted","Data":"6065935ff0a9f26bd3334a22eaa7bee531fa112d39dcc1b5042ec24b959148bf"} Nov 04 03:16:55.172836 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:55.172728 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-t2twg" podStartSLOduration=3.972416693 podStartE2EDuration="12.172710304s" totalImagesPullingTime="8.200293611s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 03:16:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:16:54.39794546 +0000 UTC m=+11.995493256" watchObservedRunningTime="2026-11-04 03:16:55.172710304 +0000 UTC m=+12.770258098" Nov 04 03:16:55.173912 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:55.173829 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-s2zcf" podStartSLOduration=3.444131183 podStartE2EDuration="12.173816828s" totalImagesPullingTime="8.729685645s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 03:16:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:16:55.172682693 +0000 UTC m=+12.770230487" watchObservedRunningTime="2026-11-04 03:16:55.173816828 +0000 UTC m=+12.771364623" Nov 04 03:16:55.930039 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:55.929855 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:16:55.930230 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:55.929959 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:16:55.930230 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:55.929966 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:16:55.930325 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:55.930223 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzg2c" podUID="1aa3f04c-b684-4426-8d46-a36d8c4e77be" Nov 04 03:16:55.930325 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:55.930112 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hjpn" podUID="719911b7-0c11-4c79-bb86-fdc043d9f9ab" Nov 04 03:16:55.930325 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:55.930271 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rmjv" podUID="834b5587-1820-4089-a925-ab992fe654b4" Nov 04 03:16:56.062072 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:56.062037 2578 generic.go:356] "Generic (PLEG): container finished" podID="96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab" containerID="9a372e85dccd7c6e891a11f276d1f9e6113c43a01ded557ad82da989d7b395fd" exitCode=0 Nov 04 03:16:56.062231 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:56.062128 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q5p48" event={"ID":"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab","Type":"ContainerDied","Data":"9a372e85dccd7c6e891a11f276d1f9e6113c43a01ded557ad82da989d7b395fd"} Nov 04 03:16:56.065068 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:56.065036 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" event={"ID":"e5e3933f-6610-46a7-934a-8ce205da98d9","Type":"ContainerStarted","Data":"4cb4740e782cad68fe16fa914f936d578d9de6c4e3182ecad7121bbd6396a753"} Nov 04 03:16:56.066684 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:56.066647 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" event={"ID":"ca5887c7-01d6-44ee-ac30-8b4374663d3e","Type":"ContainerStarted","Data":"6a1d921d047bbec5bfcaf859877929cb4167761a148a3034e8c47275f248bd5a"} Nov 04 03:16:56.300415 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:56.300359 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" podStartSLOduration=3.541827222 podStartE2EDuration="13.300345597s" totalImagesPullingTime="9.758518375s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 03:16:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:16:56.299847441 +0000 UTC m=+13.897395236" watchObservedRunningTime="2026-11-04 03:16:56.300345597 +0000 UTC m=+13.897893391" Nov 04 03:16:56.421086 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:56.421031 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" podStartSLOduration=2.6021396599999997 podStartE2EDuration="13.421015631s" totalImagesPullingTime="10.818875971s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 03:16:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:16:56.420708434 +0000 UTC m=+14.018256246" watchObservedRunningTime="2026-11-04 03:16:56.421015631 +0000 UTC m=+14.018563425" Nov 04 03:16:57.930676 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:57.930633 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:16:57.930676 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:57.930662 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:16:57.931462 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:57.930668 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:16:57.931462 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:57.930783 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzg2c" podUID="1aa3f04c-b684-4426-8d46-a36d8c4e77be" Nov 04 03:16:57.931462 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:57.930859 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hjpn" podUID="719911b7-0c11-4c79-bb86-fdc043d9f9ab" Nov 04 03:16:57.931462 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:57.930952 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rmjv" podUID="834b5587-1820-4089-a925-ab992fe654b4" Nov 04 03:16:58.075333 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:58.075300 2578 generic.go:356] "Generic (PLEG): container finished" podID="96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab" containerID="77575ea23166825ba6f7ec42146d03fc1b15d469ef0a09fb2740de06ce2d1339" exitCode=0 Nov 04 03:16:58.075499 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:58.075360 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q5p48" event={"ID":"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab","Type":"ContainerDied","Data":"77575ea23166825ba6f7ec42146d03fc1b15d469ef0a09fb2740de06ce2d1339"} Nov 04 03:16:59.854597 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:59.854382 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/834b5587-1820-4089-a925-ab992fe654b4-original-pull-secret\") pod \"global-pull-secret-syncer-8rmjv\" (UID: \"834b5587-1820-4089-a925-ab992fe654b4\") " pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:16:59.855000 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:59.854674 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aa3f04c-b684-4426-8d46-a36d8c4e77be-metrics-certs\") pod \"network-metrics-daemon-dzg2c\" (UID: \"1aa3f04c-b684-4426-8d46-a36d8c4e77be\") " pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:16:59.855000 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:59.854503 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 03:16:59.855000 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:59.854765 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/834b5587-1820-4089-a925-ab992fe654b4-original-pull-secret podName:834b5587-1820-4089-a925-ab992fe654b4 nodeName:}" failed. No retries permitted until 2026-11-04 03:17:15.85474271 +0000 UTC m=+33.452290486 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/834b5587-1820-4089-a925-ab992fe654b4-original-pull-secret") pod "global-pull-secret-syncer-8rmjv" (UID: "834b5587-1820-4089-a925-ab992fe654b4") : object "kube-system"/"original-pull-secret" not registered Nov 04 03:16:59.855000 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:59.854789 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 03:16:59.855000 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:59.854834 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aa3f04c-b684-4426-8d46-a36d8c4e77be-metrics-certs podName:1aa3f04c-b684-4426-8d46-a36d8c4e77be nodeName:}" failed. No retries permitted until 2026-11-04 03:17:15.854820655 +0000 UTC m=+33.452368441 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1aa3f04c-b684-4426-8d46-a36d8c4e77be-metrics-certs") pod "network-metrics-daemon-dzg2c" (UID: "1aa3f04c-b684-4426-8d46-a36d8c4e77be") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 03:16:59.930658 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:59.930611 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:16:59.930863 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:59.930828 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:16:59.930985 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:59.930834 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hjpn" podUID="719911b7-0c11-4c79-bb86-fdc043d9f9ab" Nov 04 03:16:59.930985 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:59.930955 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzg2c" podUID="1aa3f04c-b684-4426-8d46-a36d8c4e77be" Nov 04 03:16:59.930985 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:59.930976 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:16:59.931135 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:59.931074 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rmjv" podUID="834b5587-1820-4089-a925-ab992fe654b4" Nov 04 03:16:59.955203 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:16:59.955175 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7xg7\" (UniqueName: \"kubernetes.io/projected/719911b7-0c11-4c79-bb86-fdc043d9f9ab-kube-api-access-n7xg7\") pod \"network-check-target-2hjpn\" (UID: \"719911b7-0c11-4c79-bb86-fdc043d9f9ab\") " pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:16:59.955338 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:59.955323 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 03:16:59.955406 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:59.955342 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 03:16:59.955406 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:59.955351 2578 projected.go:196] Error preparing data for projected volume kube-api-access-n7xg7 for pod openshift-network-diagnostics/network-check-target-2hjpn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 03:16:59.955406 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:16:59.955399 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/719911b7-0c11-4c79-bb86-fdc043d9f9ab-kube-api-access-n7xg7 podName:719911b7-0c11-4c79-bb86-fdc043d9f9ab nodeName:}" failed. No retries permitted until 2026-11-04 03:17:15.955382646 +0000 UTC m=+33.552930420 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-n7xg7" (UniqueName: "kubernetes.io/projected/719911b7-0c11-4c79-bb86-fdc043d9f9ab-kube-api-access-n7xg7") pod "network-check-target-2hjpn" (UID: "719911b7-0c11-4c79-bb86-fdc043d9f9ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 03:17:00.082473 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:00.082421 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" event={"ID":"326a1269-c427-4470-86a6-70ee348c8791","Type":"ContainerStarted","Data":"f4bbd3438499977f9bf44c3aca433798a5352acf37d2fb7dcf65971832622612"} Nov 04 03:17:00.082643 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:00.082509 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" event={"ID":"326a1269-c427-4470-86a6-70ee348c8791","Type":"ContainerStarted","Data":"84e01717d009a7f0532f50ce9c92d01ab973563bfab77f9257fb8571eadc0ee2"} Nov 04 03:17:01.087806 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:01.087602 2578 generic.go:356] "Generic (PLEG): container finished" podID="96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab" containerID="f715a0a5876fb149627a0ba95b1ec978568f4d5dc3ac8a8b3e005db64aeb7906" exitCode=0 Nov 04 03:17:01.088284 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:01.087683 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q5p48" event={"ID":"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab","Type":"ContainerDied","Data":"f715a0a5876fb149627a0ba95b1ec978568f4d5dc3ac8a8b3e005db64aeb7906"} Nov 04 03:17:01.091131 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:01.091106 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" event={"ID":"326a1269-c427-4470-86a6-70ee348c8791","Type":"ContainerStarted","Data":"a752f1d598917b2985c2953d152d58fd6d3b8813e0da5ef01e9226f2e31f82d6"} Nov 04 03:17:01.091216 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:01.091170 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" event={"ID":"326a1269-c427-4470-86a6-70ee348c8791","Type":"ContainerStarted","Data":"8e1ab5cf22a63c8b03c2b59b8fa6ebe916c61fb10d0db5de6fa83157cbcd51dc"} Nov 04 03:17:01.091216 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:01.091199 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" event={"ID":"326a1269-c427-4470-86a6-70ee348c8791","Type":"ContainerStarted","Data":"3d46bf04eb2c1cfe527e46ee9144dbf32c3b41d156e1b5ab6417bb9f13d841c0"} Nov 04 03:17:01.091315 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:01.091225 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" event={"ID":"326a1269-c427-4470-86a6-70ee348c8791","Type":"ContainerStarted","Data":"9fd4712a188df7ac8a8f26ba47b8c3395ce5555e5b440422a97f418c4796b10c"} Nov 04 03:17:01.929884 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:01.929849 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:17:01.930048 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:01.929956 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:17:01.930048 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:01.929968 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hjpn" podUID="719911b7-0c11-4c79-bb86-fdc043d9f9ab" Nov 04 03:17:01.930048 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:01.929975 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:17:01.930048 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:01.930027 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rmjv" podUID="834b5587-1820-4089-a925-ab992fe654b4" Nov 04 03:17:01.930254 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:01.930101 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzg2c" podUID="1aa3f04c-b684-4426-8d46-a36d8c4e77be" Nov 04 03:17:02.096118 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:02.096081 2578 generic.go:356] "Generic (PLEG): container finished" podID="96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab" containerID="85f1612483fabbfd5b5da78ef0902893c1986baf04d13b0f3e6824419956f71f" exitCode=0 Nov 04 03:17:02.096470 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:02.096146 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q5p48" event={"ID":"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab","Type":"ContainerDied","Data":"85f1612483fabbfd5b5da78ef0902893c1986baf04d13b0f3e6824419956f71f"} Nov 04 03:17:03.101734 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:03.101689 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" event={"ID":"326a1269-c427-4470-86a6-70ee348c8791","Type":"ContainerStarted","Data":"271215718c96ac455605c4b79fdebf2799dbf6c1a51664c54b3c69f4e1c6ac85"} Nov 04 03:17:03.104806 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:03.104765 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q5p48" event={"ID":"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab","Type":"ContainerStarted","Data":"7d0dfc25eb511829a419887802f301d101193e54aa7b93be5618a6b37ed9ba76"} Nov 04 03:17:03.238299 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:03.238245 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-q5p48" podStartSLOduration=8.777790945 podStartE2EDuration="20.238230912s" totalImagesPullingTime="11.460439967s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 03:16:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:17:03.236907331 +0000 UTC m=+20.834455126" watchObservedRunningTime="2026-11-04 03:17:03.238230912 +0000 UTC m=+20.835778706" Nov 04 03:17:03.929740 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:03.929696 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:17:03.929740 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:03.929730 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:17:03.929973 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:03.929696 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:17:03.929973 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:03.929815 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hjpn" podUID="719911b7-0c11-4c79-bb86-fdc043d9f9ab" Nov 04 03:17:03.929973 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:03.929903 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rmjv" podUID="834b5587-1820-4089-a925-ab992fe654b4" Nov 04 03:17:03.930072 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:03.929973 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzg2c" podUID="1aa3f04c-b684-4426-8d46-a36d8c4e77be" Nov 04 03:17:04.005477 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:04.005438 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pmv6n" Nov 04 03:17:04.005689 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:04.005579 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 03:17:04.006077 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:04.006061 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pmv6n" Nov 04 03:17:05.114260 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:05.114224 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" event={"ID":"326a1269-c427-4470-86a6-70ee348c8791","Type":"ContainerStarted","Data":"7c207ae967ff876b9dba2bc6362d2e489f3c8975dc3d576c46a8c7a2ef525c97"} Nov 04 03:17:05.115146 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:05.114507 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:05.115146 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:05.114580 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:05.130402 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:05.130372 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:05.130534 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:05.130481 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:05.394294 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:05.393266 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" podStartSLOduration=7.85019619 podStartE2EDuration="22.393247467s" totalImagesPullingTime="14.543051277s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 03:16:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:17:05.392646342 +0000 UTC m=+22.990194131" watchObservedRunningTime="2026-11-04 03:17:05.393247467 +0000 UTC m=+22.990795264" Nov 04 03:17:05.930136 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:05.930094 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:17:05.930323 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:05.930179 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:17:05.930323 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:05.930266 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:17:05.930323 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:05.930284 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rmjv" podUID="834b5587-1820-4089-a925-ab992fe654b4" Nov 04 03:17:05.930514 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:05.930352 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzg2c" podUID="1aa3f04c-b684-4426-8d46-a36d8c4e77be" Nov 04 03:17:05.930514 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:05.930410 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hjpn" podUID="719911b7-0c11-4c79-bb86-fdc043d9f9ab" Nov 04 03:17:06.116828 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:06.116803 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 03:17:07.118783 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:07.118753 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 03:17:07.320081 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:07.320048 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r9xhm_fb15c957-99e8-4c15-b382-788025661809/dns-node-resolver/0.log" Nov 04 03:17:07.930328 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:07.930297 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:17:07.930494 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:07.930400 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hjpn" podUID="719911b7-0c11-4c79-bb86-fdc043d9f9ab" Nov 04 03:17:07.930494 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:07.930403 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:17:07.930494 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:07.930480 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzg2c" podUID="1aa3f04c-b684-4426-8d46-a36d8c4e77be" Nov 04 03:17:07.930494 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:07.930403 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:17:07.930644 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:07.930540 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rmjv" podUID="834b5587-1820-4089-a925-ab992fe654b4" Nov 04 03:17:08.145388 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:08.145355 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wj6dr_1f6b477e-a0fd-49c8-896d-c268a6c06d3f/node-ca/0.log" Nov 04 03:17:09.930106 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:09.929913 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:17:09.930644 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:09.929971 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:17:09.930644 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:09.930204 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hjpn" podUID="719911b7-0c11-4c79-bb86-fdc043d9f9ab" Nov 04 03:17:09.930644 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:09.930299 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzg2c" podUID="1aa3f04c-b684-4426-8d46-a36d8c4e77be" Nov 04 03:17:09.930644 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:09.929978 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:17:09.930644 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:09.930401 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rmjv" podUID="834b5587-1820-4089-a925-ab992fe654b4" Nov 04 03:17:11.443694 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:11.443651 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:11.444088 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:11.443852 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 03:17:11.461021 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:11.460977 2578 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" podUID="326a1269-c427-4470-86a6-70ee348c8791" containerName="ovnkube-controller" probeResult="failure" output="" Nov 04 03:17:11.469706 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:11.469680 2578 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" podUID="326a1269-c427-4470-86a6-70ee348c8791" containerName="ovnkube-controller" probeResult="failure" output="" Nov 04 03:17:11.930274 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:11.930239 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:17:11.930274 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:11.930272 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:17:11.930495 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:11.930376 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rmjv" podUID="834b5587-1820-4089-a925-ab992fe654b4" Nov 04 03:17:11.930495 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:11.930450 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:17:11.930569 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:11.930533 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzg2c" podUID="1aa3f04c-b684-4426-8d46-a36d8c4e77be" Nov 04 03:17:11.930639 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:11.930590 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hjpn" podUID="719911b7-0c11-4c79-bb86-fdc043d9f9ab" Nov 04 03:17:13.930396 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:13.930359 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:17:13.930875 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:13.930438 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:17:13.930875 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:13.930523 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:17:13.930875 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:13.930545 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzg2c" podUID="1aa3f04c-b684-4426-8d46-a36d8c4e77be" Nov 04 03:17:13.930875 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:13.930585 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hjpn" podUID="719911b7-0c11-4c79-bb86-fdc043d9f9ab" Nov 04 03:17:13.930875 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:13.930709 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rmjv" podUID="834b5587-1820-4089-a925-ab992fe654b4" Nov 04 03:17:15.874080 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:15.874042 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aa3f04c-b684-4426-8d46-a36d8c4e77be-metrics-certs\") pod \"network-metrics-daemon-dzg2c\" (UID: \"1aa3f04c-b684-4426-8d46-a36d8c4e77be\") " pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:17:15.874080 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:15.874089 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/834b5587-1820-4089-a925-ab992fe654b4-original-pull-secret\") pod \"global-pull-secret-syncer-8rmjv\" (UID: \"834b5587-1820-4089-a925-ab992fe654b4\") " pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:17:15.874550 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:15.874188 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 03:17:15.874550 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:15.874201 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 03:17:15.874550 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:15.874245 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/834b5587-1820-4089-a925-ab992fe654b4-original-pull-secret podName:834b5587-1820-4089-a925-ab992fe654b4 nodeName:}" failed. No retries permitted until 2026-11-04 03:17:47.874229506 +0000 UTC m=+65.471777278 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/834b5587-1820-4089-a925-ab992fe654b4-original-pull-secret") pod "global-pull-secret-syncer-8rmjv" (UID: "834b5587-1820-4089-a925-ab992fe654b4") : object "kube-system"/"original-pull-secret" not registered Nov 04 03:17:15.874550 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:15.874272 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aa3f04c-b684-4426-8d46-a36d8c4e77be-metrics-certs podName:1aa3f04c-b684-4426-8d46-a36d8c4e77be nodeName:}" failed. No retries permitted until 2026-11-04 03:17:47.874252723 +0000 UTC m=+65.471800497 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1aa3f04c-b684-4426-8d46-a36d8c4e77be-metrics-certs") pod "network-metrics-daemon-dzg2c" (UID: "1aa3f04c-b684-4426-8d46-a36d8c4e77be") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 03:17:15.930069 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:15.930032 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:17:15.930069 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:15.930056 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:17:15.930264 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:15.930033 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:17:15.930264 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:15.930153 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rmjv" podUID="834b5587-1820-4089-a925-ab992fe654b4" Nov 04 03:17:15.930368 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:15.930256 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzg2c" podUID="1aa3f04c-b684-4426-8d46-a36d8c4e77be" Nov 04 03:17:15.930368 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:15.930346 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hjpn" podUID="719911b7-0c11-4c79-bb86-fdc043d9f9ab" Nov 04 03:17:15.974442 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:15.974403 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7xg7\" (UniqueName: \"kubernetes.io/projected/719911b7-0c11-4c79-bb86-fdc043d9f9ab-kube-api-access-n7xg7\") pod \"network-check-target-2hjpn\" (UID: \"719911b7-0c11-4c79-bb86-fdc043d9f9ab\") " pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:17:15.974612 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:15.974558 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 03:17:15.974612 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:15.974580 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 03:17:15.974612 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:15.974589 2578 projected.go:196] Error preparing data for projected volume kube-api-access-n7xg7 for pod openshift-network-diagnostics/network-check-target-2hjpn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 03:17:15.974731 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:15.974655 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/719911b7-0c11-4c79-bb86-fdc043d9f9ab-kube-api-access-n7xg7 podName:719911b7-0c11-4c79-bb86-fdc043d9f9ab nodeName:}" failed. No retries permitted until 2026-11-04 03:17:47.974641465 +0000 UTC m=+65.572189249 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-n7xg7" (UniqueName: "kubernetes.io/projected/719911b7-0c11-4c79-bb86-fdc043d9f9ab-kube-api-access-n7xg7") pod "network-check-target-2hjpn" (UID: "719911b7-0c11-4c79-bb86-fdc043d9f9ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 03:17:17.930495 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:17.930448 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:17:17.930994 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:17.930550 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:17:17.930994 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:17.930555 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hjpn" podUID="719911b7-0c11-4c79-bb86-fdc043d9f9ab" Nov 04 03:17:17.930994 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:17.930560 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:17:17.930994 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:17.930686 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rmjv" podUID="834b5587-1820-4089-a925-ab992fe654b4" Nov 04 03:17:17.930994 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:17.930788 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzg2c" podUID="1aa3f04c-b684-4426-8d46-a36d8c4e77be" Nov 04 03:17:19.930692 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:19.930645 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:17:19.930692 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:19.930696 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:17:19.931168 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:19.930655 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:17:19.931168 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:19.930781 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzg2c" podUID="1aa3f04c-b684-4426-8d46-a36d8c4e77be" Nov 04 03:17:19.931168 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:19.930821 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hjpn" podUID="719911b7-0c11-4c79-bb86-fdc043d9f9ab" Nov 04 03:17:19.931168 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:19.930889 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rmjv" podUID="834b5587-1820-4089-a925-ab992fe654b4" Nov 04 03:17:21.930245 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:21.930213 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:17:21.930745 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:21.930320 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:17:21.930745 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:21.930332 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hjpn" podUID="719911b7-0c11-4c79-bb86-fdc043d9f9ab" Nov 04 03:17:21.930745 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:21.930402 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rmjv" podUID="834b5587-1820-4089-a925-ab992fe654b4" Nov 04 03:17:21.930745 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:21.930407 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:17:21.930745 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:21.930474 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzg2c" podUID="1aa3f04c-b684-4426-8d46-a36d8c4e77be" Nov 04 03:17:23.930643 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:23.930583 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:17:23.930643 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:23.930602 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:17:23.931116 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:23.930704 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:17:23.931116 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:23.930718 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rmjv" podUID="834b5587-1820-4089-a925-ab992fe654b4" Nov 04 03:17:23.931116 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:23.930781 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzg2c" podUID="1aa3f04c-b684-4426-8d46-a36d8c4e77be" Nov 04 03:17:23.931116 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:23.930864 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hjpn" podUID="719911b7-0c11-4c79-bb86-fdc043d9f9ab" Nov 04 03:17:25.930042 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:25.930001 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:17:25.930042 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:25.930040 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:17:25.930543 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:25.930065 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:17:25.930543 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:25.930164 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzg2c" podUID="1aa3f04c-b684-4426-8d46-a36d8c4e77be" Nov 04 03:17:25.930543 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:25.930275 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hjpn" podUID="719911b7-0c11-4c79-bb86-fdc043d9f9ab" Nov 04 03:17:25.930543 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:25.930334 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rmjv" podUID="834b5587-1820-4089-a925-ab992fe654b4" Nov 04 03:17:27.162199 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:27.162162 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8rmjv"] Nov 04 03:17:27.162685 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:27.162295 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:17:27.162685 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:27.162415 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rmjv" podUID="834b5587-1820-4089-a925-ab992fe654b4" Nov 04 03:17:27.166496 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:27.166466 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2hjpn"] Nov 04 03:17:27.166669 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:27.166524 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dzg2c"] Nov 04 03:17:27.166669 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:27.166589 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:17:27.166792 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:27.166710 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzg2c" podUID="1aa3f04c-b684-4426-8d46-a36d8c4e77be" Nov 04 03:17:27.166884 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:27.166870 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:17:27.166961 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:27.166943 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hjpn" podUID="719911b7-0c11-4c79-bb86-fdc043d9f9ab" Nov 04 03:17:28.930170 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:28.930088 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:17:28.930170 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:28.930123 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:17:28.930579 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:28.930095 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:17:28.930579 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:28.930218 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rmjv" podUID="834b5587-1820-4089-a925-ab992fe654b4" Nov 04 03:17:28.930579 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:28.930295 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzg2c" podUID="1aa3f04c-b684-4426-8d46-a36d8c4e77be" Nov 04 03:17:28.930579 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:28.930366 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hjpn" podUID="719911b7-0c11-4c79-bb86-fdc043d9f9ab" Nov 04 03:17:30.929677 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:30.929642 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:17:30.929677 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:30.929649 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:17:30.930109 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:30.929795 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dzg2c" podUID="1aa3f04c-b684-4426-8d46-a36d8c4e77be" Nov 04 03:17:30.930109 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:30.929832 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:17:30.930109 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:30.929896 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hjpn" podUID="719911b7-0c11-4c79-bb86-fdc043d9f9ab" Nov 04 03:17:30.930109 ip-10-0-133-6 kubenswrapper[2578]: E1104 03:17:30.929955 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rmjv" podUID="834b5587-1820-4089-a925-ab992fe654b4" Nov 04 03:17:31.678297 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:31.678271 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-6.us-west-1.compute.internal" event="NodeReady" Nov 04 03:17:31.678474 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:31.678388 2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 04 03:17:31.836404 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:31.836369 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jg2pw"] Nov 04 03:17:31.850779 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:31.850754 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wv6zh"] Nov 04 03:17:31.850884 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:31.850871 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jg2pw" Nov 04 03:17:31.863380 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:31.863362 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wv6zh" Nov 04 03:17:31.912752 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:31.912725 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-p8x4k"] Nov 04 03:17:31.917403 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:31.917382 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 04 03:17:31.917899 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:31.917885 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n89nd\"" Nov 04 03:17:31.923154 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:31.923138 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p8x4k" Nov 04 03:17:31.924377 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:31.924355 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jg2pw"] Nov 04 03:17:31.932126 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:31.932098 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wv6zh"] Nov 04 03:17:31.985966 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:31.985936 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44f960f8-1c9c-4a1b-a422-5477a6d49316-cert\") pod \"ingress-canary-wv6zh\" (UID: \"44f960f8-1c9c-4a1b-a422-5477a6d49316\") " pod="openshift-ingress-canary/ingress-canary-wv6zh" Nov 04 03:17:31.986119 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:31.985976 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s869g\" (UniqueName: \"kubernetes.io/projected/9acbca43-6baa-4f8f-9eee-eb08dc731588-kube-api-access-s869g\") pod \"dns-default-jg2pw\" (UID: \"9acbca43-6baa-4f8f-9eee-eb08dc731588\") " pod="openshift-dns/dns-default-jg2pw" Nov 04 03:17:31.986119 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:31.986005 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szld4\" (UniqueName: \"kubernetes.io/projected/44f960f8-1c9c-4a1b-a422-5477a6d49316-kube-api-access-szld4\") pod \"ingress-canary-wv6zh\" (UID: \"44f960f8-1c9c-4a1b-a422-5477a6d49316\") " pod="openshift-ingress-canary/ingress-canary-wv6zh" Nov 04 03:17:31.986119 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:31.986061 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9acbca43-6baa-4f8f-9eee-eb08dc731588-metrics-tls\") pod \"dns-default-jg2pw\" (UID: \"9acbca43-6baa-4f8f-9eee-eb08dc731588\") " pod="openshift-dns/dns-default-jg2pw" Nov 04 03:17:31.986228 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:31.986213 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9acbca43-6baa-4f8f-9eee-eb08dc731588-tmp-dir\") pod \"dns-default-jg2pw\" (UID: \"9acbca43-6baa-4f8f-9eee-eb08dc731588\") " pod="openshift-dns/dns-default-jg2pw" Nov 04 03:17:31.986264 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:31.986244 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9acbca43-6baa-4f8f-9eee-eb08dc731588-config-volume\") pod \"dns-default-jg2pw\" (UID: \"9acbca43-6baa-4f8f-9eee-eb08dc731588\") " pod="openshift-dns/dns-default-jg2pw" Nov 04 03:17:31.999473 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:31.999442 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p8x4k"] Nov 04 03:17:32.022522 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.022490 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 04 03:17:32.022675 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.022537 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 04 03:17:32.022675 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.022602 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 04 03:17:32.022840 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.022757 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 04 03:17:32.022840 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.022772 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xmpr4\"" Nov 04 03:17:32.022840 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.022791 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 04 03:17:32.022840 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.022811 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 04 03:17:32.022840 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.022820 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 04 03:17:32.022998 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.022846 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-2qrjk\"" Nov 04 03:17:32.022998 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.022850 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 04 03:17:32.087360 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.087336 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9acbca43-6baa-4f8f-9eee-eb08dc731588-tmp-dir\") pod \"dns-default-jg2pw\" (UID: \"9acbca43-6baa-4f8f-9eee-eb08dc731588\") " pod="openshift-dns/dns-default-jg2pw" Nov 04 03:17:32.087453 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.087373 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c63fa297-f5dd-4c87-aa7b-f691ad61cb45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p8x4k\" (UID: \"c63fa297-f5dd-4c87-aa7b-f691ad61cb45\") " pod="openshift-insights/insights-runtime-extractor-p8x4k" Nov 04 03:17:32.087453 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.087413 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c63fa297-f5dd-4c87-aa7b-f691ad61cb45-crio-socket\") pod \"insights-runtime-extractor-p8x4k\" (UID: \"c63fa297-f5dd-4c87-aa7b-f691ad61cb45\") " pod="openshift-insights/insights-runtime-extractor-p8x4k" Nov 04 03:17:32.087517 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.087482 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9acbca43-6baa-4f8f-9eee-eb08dc731588-config-volume\") pod \"dns-default-jg2pw\" (UID: \"9acbca43-6baa-4f8f-9eee-eb08dc731588\") " pod="openshift-dns/dns-default-jg2pw" Nov 04 03:17:32.087615 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.087587 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c63fa297-f5dd-4c87-aa7b-f691ad61cb45-data-volume\") pod \"insights-runtime-extractor-p8x4k\" (UID: \"c63fa297-f5dd-4c87-aa7b-f691ad61cb45\") " pod="openshift-insights/insights-runtime-extractor-p8x4k" Nov 04 03:17:32.087681 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.087667 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c63fa297-f5dd-4c87-aa7b-f691ad61cb45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p8x4k\" (UID: \"c63fa297-f5dd-4c87-aa7b-f691ad61cb45\") " pod="openshift-insights/insights-runtime-extractor-p8x4k" Nov 04 03:17:32.087786 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.087765 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2ftw\" (UniqueName: \"kubernetes.io/projected/c63fa297-f5dd-4c87-aa7b-f691ad61cb45-kube-api-access-j2ftw\") pod \"insights-runtime-extractor-p8x4k\" (UID: \"c63fa297-f5dd-4c87-aa7b-f691ad61cb45\") " pod="openshift-insights/insights-runtime-extractor-p8x4k" Nov 04 03:17:32.087833 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.087803 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44f960f8-1c9c-4a1b-a422-5477a6d49316-cert\") pod \"ingress-canary-wv6zh\" (UID: \"44f960f8-1c9c-4a1b-a422-5477a6d49316\") " pod="openshift-ingress-canary/ingress-canary-wv6zh" Nov 04 03:17:32.087866 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.087829 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s869g\" (UniqueName: \"kubernetes.io/projected/9acbca43-6baa-4f8f-9eee-eb08dc731588-kube-api-access-s869g\") pod \"dns-default-jg2pw\" (UID: \"9acbca43-6baa-4f8f-9eee-eb08dc731588\") " pod="openshift-dns/dns-default-jg2pw" Nov 04 03:17:32.087866 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.087854 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szld4\" (UniqueName: \"kubernetes.io/projected/44f960f8-1c9c-4a1b-a422-5477a6d49316-kube-api-access-szld4\") pod \"ingress-canary-wv6zh\" (UID: \"44f960f8-1c9c-4a1b-a422-5477a6d49316\") " pod="openshift-ingress-canary/ingress-canary-wv6zh" Nov 04 03:17:32.087959 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.087881 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9acbca43-6baa-4f8f-9eee-eb08dc731588-metrics-tls\") pod \"dns-default-jg2pw\" (UID: \"9acbca43-6baa-4f8f-9eee-eb08dc731588\") " pod="openshift-dns/dns-default-jg2pw" Nov 04 03:17:32.088032 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.088017 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9acbca43-6baa-4f8f-9eee-eb08dc731588-tmp-dir\") pod \"dns-default-jg2pw\" (UID: \"9acbca43-6baa-4f8f-9eee-eb08dc731588\") " pod="openshift-dns/dns-default-jg2pw" Nov 04 03:17:32.088255 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.088238 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9acbca43-6baa-4f8f-9eee-eb08dc731588-config-volume\") pod \"dns-default-jg2pw\" (UID: \"9acbca43-6baa-4f8f-9eee-eb08dc731588\") " pod="openshift-dns/dns-default-jg2pw" Nov 04 03:17:32.091593 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.091565 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9acbca43-6baa-4f8f-9eee-eb08dc731588-metrics-tls\") pod \"dns-default-jg2pw\" (UID: \"9acbca43-6baa-4f8f-9eee-eb08dc731588\") " pod="openshift-dns/dns-default-jg2pw" Nov 04 03:17:32.091705 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.091645 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44f960f8-1c9c-4a1b-a422-5477a6d49316-cert\") pod \"ingress-canary-wv6zh\" (UID: \"44f960f8-1c9c-4a1b-a422-5477a6d49316\") " pod="openshift-ingress-canary/ingress-canary-wv6zh" Nov 04 03:17:32.151288 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.151253 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s869g\" (UniqueName: \"kubernetes.io/projected/9acbca43-6baa-4f8f-9eee-eb08dc731588-kube-api-access-s869g\") pod \"dns-default-jg2pw\" (UID: \"9acbca43-6baa-4f8f-9eee-eb08dc731588\") " pod="openshift-dns/dns-default-jg2pw" Nov 04 03:17:32.151394 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.151377 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szld4\" (UniqueName: \"kubernetes.io/projected/44f960f8-1c9c-4a1b-a422-5477a6d49316-kube-api-access-szld4\") pod \"ingress-canary-wv6zh\" (UID: \"44f960f8-1c9c-4a1b-a422-5477a6d49316\") " pod="openshift-ingress-canary/ingress-canary-wv6zh" Nov 04 03:17:32.178732 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.178714 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jg2pw" Nov 04 03:17:32.188981 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.188938 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2ftw\" (UniqueName: \"kubernetes.io/projected/c63fa297-f5dd-4c87-aa7b-f691ad61cb45-kube-api-access-j2ftw\") pod \"insights-runtime-extractor-p8x4k\" (UID: \"c63fa297-f5dd-4c87-aa7b-f691ad61cb45\") " pod="openshift-insights/insights-runtime-extractor-p8x4k" Nov 04 03:17:32.189051 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.188987 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c63fa297-f5dd-4c87-aa7b-f691ad61cb45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p8x4k\" (UID: \"c63fa297-f5dd-4c87-aa7b-f691ad61cb45\") " pod="openshift-insights/insights-runtime-extractor-p8x4k" Nov 04 03:17:32.189124 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.189101 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c63fa297-f5dd-4c87-aa7b-f691ad61cb45-crio-socket\") pod \"insights-runtime-extractor-p8x4k\" (UID: \"c63fa297-f5dd-4c87-aa7b-f691ad61cb45\") " pod="openshift-insights/insights-runtime-extractor-p8x4k" Nov 04 03:17:32.189165 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.189152 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c63fa297-f5dd-4c87-aa7b-f691ad61cb45-data-volume\") pod \"insights-runtime-extractor-p8x4k\" (UID: \"c63fa297-f5dd-4c87-aa7b-f691ad61cb45\") " pod="openshift-insights/insights-runtime-extractor-p8x4k" Nov 04 03:17:32.189218 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.189195 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c63fa297-f5dd-4c87-aa7b-f691ad61cb45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p8x4k\" (UID: \"c63fa297-f5dd-4c87-aa7b-f691ad61cb45\") " pod="openshift-insights/insights-runtime-extractor-p8x4k" Nov 04 03:17:32.189340 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.189320 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c63fa297-f5dd-4c87-aa7b-f691ad61cb45-crio-socket\") pod \"insights-runtime-extractor-p8x4k\" (UID: \"c63fa297-f5dd-4c87-aa7b-f691ad61cb45\") " pod="openshift-insights/insights-runtime-extractor-p8x4k" Nov 04 03:17:32.189501 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.189483 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c63fa297-f5dd-4c87-aa7b-f691ad61cb45-data-volume\") pod \"insights-runtime-extractor-p8x4k\" (UID: \"c63fa297-f5dd-4c87-aa7b-f691ad61cb45\") " pod="openshift-insights/insights-runtime-extractor-p8x4k" Nov 04 03:17:32.189717 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.189703 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c63fa297-f5dd-4c87-aa7b-f691ad61cb45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p8x4k\" (UID: \"c63fa297-f5dd-4c87-aa7b-f691ad61cb45\") " pod="openshift-insights/insights-runtime-extractor-p8x4k" Nov 04 03:17:32.191020 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.190998 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c63fa297-f5dd-4c87-aa7b-f691ad61cb45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p8x4k\" (UID: \"c63fa297-f5dd-4c87-aa7b-f691ad61cb45\") " pod="openshift-insights/insights-runtime-extractor-p8x4k" Nov 04 03:17:32.191080 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.191070 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wv6zh" Nov 04 03:17:32.249789 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.249750 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2ftw\" (UniqueName: \"kubernetes.io/projected/c63fa297-f5dd-4c87-aa7b-f691ad61cb45-kube-api-access-j2ftw\") pod \"insights-runtime-extractor-p8x4k\" (UID: \"c63fa297-f5dd-4c87-aa7b-f691ad61cb45\") " pod="openshift-insights/insights-runtime-extractor-p8x4k" Nov 04 03:17:32.260832 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.260722 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p8x4k" Nov 04 03:17:32.584864 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.584825 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wv6zh"] Nov 04 03:17:32.588958 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:17:32.588930 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44f960f8_1c9c_4a1b_a422_5477a6d49316.slice/crio-d5c3b5befed935528d1a226b03c5e57dd4e4f5bee495cdf947cc098a3657f302 WatchSource:0}: Error finding container d5c3b5befed935528d1a226b03c5e57dd4e4f5bee495cdf947cc098a3657f302: Status 404 returned error can't find the container with id d5c3b5befed935528d1a226b03c5e57dd4e4f5bee495cdf947cc098a3657f302 Nov 04 03:17:32.635486 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.635415 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jg2pw"] Nov 04 03:17:32.638911 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.638740 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p8x4k"] Nov 04 03:17:32.639629 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:17:32.639588 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9acbca43_6baa_4f8f_9eee_eb08dc731588.slice/crio-1a6cc4f32dbf89779adc13674239ca74ae1dd88dd77759d62708826570f184b4 WatchSource:0}: Error finding container 1a6cc4f32dbf89779adc13674239ca74ae1dd88dd77759d62708826570f184b4: Status 404 returned error can't find the container with id 1a6cc4f32dbf89779adc13674239ca74ae1dd88dd77759d62708826570f184b4 Nov 04 03:17:32.643048 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:17:32.643026 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc63fa297_f5dd_4c87_aa7b_f691ad61cb45.slice/crio-a54ba8507a55dea18a87dbe7af55c0f565972c855b2bf5b1b97864bc36355c1d WatchSource:0}: Error finding container a54ba8507a55dea18a87dbe7af55c0f565972c855b2bf5b1b97864bc36355c1d: Status 404 returned error can't find the container with id a54ba8507a55dea18a87dbe7af55c0f565972c855b2bf5b1b97864bc36355c1d Nov 04 03:17:32.933253 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.933171 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:17:32.933771 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.933276 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:17:32.933771 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.933316 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:17:32.986207 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.986185 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x9qdq\"" Nov 04 03:17:32.986463 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.986440 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 04 03:17:32.986805 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.986786 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-kfskp\"" Nov 04 03:17:32.986884 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.986815 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-f24jt\"" Nov 04 03:17:32.986884 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.986865 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 04 03:17:32.986884 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.986879 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 04 03:17:32.986997 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:32.986985 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 04 03:17:33.177819 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:33.177790 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wv6zh" event={"ID":"44f960f8-1c9c-4a1b-a422-5477a6d49316","Type":"ContainerStarted","Data":"d5c3b5befed935528d1a226b03c5e57dd4e4f5bee495cdf947cc098a3657f302"} Nov 04 03:17:33.179252 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:33.179231 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p8x4k" event={"ID":"c63fa297-f5dd-4c87-aa7b-f691ad61cb45","Type":"ContainerStarted","Data":"97ae05601ba17bbf4c92df2cde8ec9168c71c7df4988ba6f5e96b0ab340911fe"} Nov 04 03:17:33.179370 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:33.179288 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p8x4k" event={"ID":"c63fa297-f5dd-4c87-aa7b-f691ad61cb45","Type":"ContainerStarted","Data":"a54ba8507a55dea18a87dbe7af55c0f565972c855b2bf5b1b97864bc36355c1d"} Nov 04 03:17:33.180303 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:33.180277 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jg2pw" event={"ID":"9acbca43-6baa-4f8f-9eee-eb08dc731588","Type":"ContainerStarted","Data":"1a6cc4f32dbf89779adc13674239ca74ae1dd88dd77759d62708826570f184b4"} Nov 04 03:17:34.548396 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.548358 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-jf99x"] Nov 04 03:17:34.550449 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.550423 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jf99x" Nov 04 03:17:34.603182 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.603154 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 04 03:17:34.603327 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.603287 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 04 03:17:34.603327 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.603320 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 04 03:17:34.603445 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.603404 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 04 03:17:34.603445 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.603327 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 04 03:17:34.603445 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.603404 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-sslkp\"" Nov 04 03:17:34.610293 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.610268 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-jf99x"] Nov 04 03:17:34.651229 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.650315 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc"] Nov 04 03:17:34.652442 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.652421 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc" Nov 04 03:17:34.673074 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.673050 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7c9tl"] Nov 04 03:17:34.675074 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.675043 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:34.705306 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.705282 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 04 03:17:34.705587 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.705537 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-tmmps\"" Nov 04 03:17:34.705587 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.705571 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 04 03:17:34.705750 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.705573 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 04 03:17:34.709924 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.709901 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a040e5d8-7316-4a37-a045-9eb7c2eaf607-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-jf99x\" (UID: \"a040e5d8-7316-4a37-a045-9eb7c2eaf607\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jf99x" Nov 04 03:17:34.710078 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.709945 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a040e5d8-7316-4a37-a045-9eb7c2eaf607-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-jf99x\" (UID: \"a040e5d8-7316-4a37-a045-9eb7c2eaf607\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jf99x" Nov 04 03:17:34.710078 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.709968 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvm8t\" (UniqueName: \"kubernetes.io/projected/a040e5d8-7316-4a37-a045-9eb7c2eaf607-kube-api-access-pvm8t\") pod \"openshift-state-metrics-7bd7489957-jf99x\" (UID: \"a040e5d8-7316-4a37-a045-9eb7c2eaf607\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jf99x" Nov 04 03:17:34.710183 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.710112 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a040e5d8-7316-4a37-a045-9eb7c2eaf607-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-jf99x\" (UID: \"a040e5d8-7316-4a37-a045-9eb7c2eaf607\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jf99x" Nov 04 03:17:34.713754 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.713734 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc"] Nov 04 03:17:34.737882 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.737854 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 04 03:17:34.737994 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.737882 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-krstf\"" Nov 04 03:17:34.737994 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.737903 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 04 03:17:34.805978 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.805891 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 04 03:17:34.811095 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.811072 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a040e5d8-7316-4a37-a045-9eb7c2eaf607-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-jf99x\" (UID: \"a040e5d8-7316-4a37-a045-9eb7c2eaf607\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jf99x" Nov 04 03:17:34.811229 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.811105 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/57272665-8b2d-4854-bcb6-98f662716f4c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-ztrlc\" (UID: \"57272665-8b2d-4854-bcb6-98f662716f4c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc" Nov 04 03:17:34.811229 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.811138 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/57272665-8b2d-4854-bcb6-98f662716f4c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-ztrlc\" (UID: \"57272665-8b2d-4854-bcb6-98f662716f4c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc" Nov 04 03:17:34.811229 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.811177 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7c4r\" (UniqueName: \"kubernetes.io/projected/57272665-8b2d-4854-bcb6-98f662716f4c-kube-api-access-s7c4r\") pod \"kube-state-metrics-5dc8d7d89b-ztrlc\" (UID: \"57272665-8b2d-4854-bcb6-98f662716f4c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc" Nov 04 03:17:34.811396 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.811237 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a040e5d8-7316-4a37-a045-9eb7c2eaf607-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-jf99x\" (UID: \"a040e5d8-7316-4a37-a045-9eb7c2eaf607\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jf99x" Nov 04 03:17:34.811658 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.811633 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/16659cac-4c32-4a94-90aa-8be0c6be33b2-node-exporter-accelerators-collector-config\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:34.811779 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.811708 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16659cac-4c32-4a94-90aa-8be0c6be33b2-sys\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:34.811779 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.811732 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/57272665-8b2d-4854-bcb6-98f662716f4c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-ztrlc\" (UID: \"57272665-8b2d-4854-bcb6-98f662716f4c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc" Nov 04 03:17:34.811779 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.811751 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/16659cac-4c32-4a94-90aa-8be0c6be33b2-node-exporter-textfile\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:34.811936 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.811834 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16659cac-4c32-4a94-90aa-8be0c6be33b2-metrics-client-ca\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:34.811936 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.811861 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16659cac-4c32-4a94-90aa-8be0c6be33b2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:34.811936 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.811895 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a040e5d8-7316-4a37-a045-9eb7c2eaf607-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-jf99x\" (UID: \"a040e5d8-7316-4a37-a045-9eb7c2eaf607\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jf99x" Nov 04 03:17:34.811936 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.811912 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvm8t\" (UniqueName: \"kubernetes.io/projected/a040e5d8-7316-4a37-a045-9eb7c2eaf607-kube-api-access-pvm8t\") pod \"openshift-state-metrics-7bd7489957-jf99x\" (UID: \"a040e5d8-7316-4a37-a045-9eb7c2eaf607\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jf99x" Nov 04 03:17:34.811936 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.811929 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/16659cac-4c32-4a94-90aa-8be0c6be33b2-root\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:34.811936 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.811929 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a040e5d8-7316-4a37-a045-9eb7c2eaf607-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-jf99x\" (UID: \"a040e5d8-7316-4a37-a045-9eb7c2eaf607\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jf99x" Nov 04 03:17:34.812106 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.811985 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/16659cac-4c32-4a94-90aa-8be0c6be33b2-node-exporter-tls\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:34.812106 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.812044 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57272665-8b2d-4854-bcb6-98f662716f4c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-ztrlc\" (UID: \"57272665-8b2d-4854-bcb6-98f662716f4c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc" Nov 04 03:17:34.812106 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.812080 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/57272665-8b2d-4854-bcb6-98f662716f4c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-ztrlc\" (UID: \"57272665-8b2d-4854-bcb6-98f662716f4c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc" Nov 04 03:17:34.812196 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.812142 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/16659cac-4c32-4a94-90aa-8be0c6be33b2-node-exporter-wtmp\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:34.812227 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.812199 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcpzg\" (UniqueName: \"kubernetes.io/projected/16659cac-4c32-4a94-90aa-8be0c6be33b2-kube-api-access-fcpzg\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:34.814314 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.814292 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a040e5d8-7316-4a37-a045-9eb7c2eaf607-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-jf99x\" (UID: \"a040e5d8-7316-4a37-a045-9eb7c2eaf607\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jf99x" Nov 04 03:17:34.816310 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.816210 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a040e5d8-7316-4a37-a045-9eb7c2eaf607-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-jf99x\" (UID: \"a040e5d8-7316-4a37-a045-9eb7c2eaf607\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jf99x" Nov 04 03:17:34.874652 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.873603 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvm8t\" (UniqueName: \"kubernetes.io/projected/a040e5d8-7316-4a37-a045-9eb7c2eaf607-kube-api-access-pvm8t\") pod \"openshift-state-metrics-7bd7489957-jf99x\" (UID: \"a040e5d8-7316-4a37-a045-9eb7c2eaf607\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jf99x" Nov 04 03:17:34.913317 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.913283 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16659cac-4c32-4a94-90aa-8be0c6be33b2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:34.913478 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.913339 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/16659cac-4c32-4a94-90aa-8be0c6be33b2-root\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:34.913478 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.913380 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/16659cac-4c32-4a94-90aa-8be0c6be33b2-node-exporter-tls\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:34.913478 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.913408 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57272665-8b2d-4854-bcb6-98f662716f4c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-ztrlc\" (UID: \"57272665-8b2d-4854-bcb6-98f662716f4c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc" Nov 04 03:17:34.913478 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.913432 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/57272665-8b2d-4854-bcb6-98f662716f4c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-ztrlc\" (UID: \"57272665-8b2d-4854-bcb6-98f662716f4c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc" Nov 04 03:17:34.913478 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.913459 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/16659cac-4c32-4a94-90aa-8be0c6be33b2-node-exporter-wtmp\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:34.913478 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.913477 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcpzg\" (UniqueName: \"kubernetes.io/projected/16659cac-4c32-4a94-90aa-8be0c6be33b2-kube-api-access-fcpzg\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:34.913797 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.913481 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/16659cac-4c32-4a94-90aa-8be0c6be33b2-root\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:34.913797 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.913659 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/16659cac-4c32-4a94-90aa-8be0c6be33b2-node-exporter-wtmp\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:34.913797 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.913780 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/57272665-8b2d-4854-bcb6-98f662716f4c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-ztrlc\" (UID: \"57272665-8b2d-4854-bcb6-98f662716f4c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc" Nov 04 03:17:34.913942 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.913829 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/57272665-8b2d-4854-bcb6-98f662716f4c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-ztrlc\" (UID: \"57272665-8b2d-4854-bcb6-98f662716f4c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc" Nov 04 03:17:34.913942 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.913869 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7c4r\" (UniqueName: \"kubernetes.io/projected/57272665-8b2d-4854-bcb6-98f662716f4c-kube-api-access-s7c4r\") pod \"kube-state-metrics-5dc8d7d89b-ztrlc\" (UID: \"57272665-8b2d-4854-bcb6-98f662716f4c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc" Nov 04 03:17:34.913942 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.913898 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/16659cac-4c32-4a94-90aa-8be0c6be33b2-node-exporter-accelerators-collector-config\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:34.913942 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.913933 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16659cac-4c32-4a94-90aa-8be0c6be33b2-sys\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:34.914132 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.913956 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/57272665-8b2d-4854-bcb6-98f662716f4c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-ztrlc\" (UID: \"57272665-8b2d-4854-bcb6-98f662716f4c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc" Nov 04 03:17:34.914132 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.913990 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/16659cac-4c32-4a94-90aa-8be0c6be33b2-node-exporter-textfile\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:34.914132 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.914067 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16659cac-4c32-4a94-90aa-8be0c6be33b2-metrics-client-ca\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:34.914442 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.914250 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57272665-8b2d-4854-bcb6-98f662716f4c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-ztrlc\" (UID: \"57272665-8b2d-4854-bcb6-98f662716f4c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc" Nov 04 03:17:34.914442 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.914382 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/57272665-8b2d-4854-bcb6-98f662716f4c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-ztrlc\" (UID: \"57272665-8b2d-4854-bcb6-98f662716f4c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc" Nov 04 03:17:34.914749 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.914689 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/16659cac-4c32-4a94-90aa-8be0c6be33b2-node-exporter-textfile\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:34.914749 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.914739 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/16659cac-4c32-4a94-90aa-8be0c6be33b2-node-exporter-accelerators-collector-config\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:34.914884 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.914770 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16659cac-4c32-4a94-90aa-8be0c6be33b2-metrics-client-ca\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:34.914884 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.914781 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16659cac-4c32-4a94-90aa-8be0c6be33b2-sys\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:34.915092 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.915058 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/57272665-8b2d-4854-bcb6-98f662716f4c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-ztrlc\" (UID: \"57272665-8b2d-4854-bcb6-98f662716f4c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc" Nov 04 03:17:34.916215 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.916195 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16659cac-4c32-4a94-90aa-8be0c6be33b2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:34.917091 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.917044 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/57272665-8b2d-4854-bcb6-98f662716f4c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-ztrlc\" (UID: \"57272665-8b2d-4854-bcb6-98f662716f4c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc" Nov 04 03:17:34.917671 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.917647 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/16659cac-4c32-4a94-90aa-8be0c6be33b2-node-exporter-tls\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:34.919879 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:34.919849 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/57272665-8b2d-4854-bcb6-98f662716f4c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-ztrlc\" (UID: \"57272665-8b2d-4854-bcb6-98f662716f4c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc" Nov 04 03:17:35.000424 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:35.000388 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcpzg\" (UniqueName: \"kubernetes.io/projected/16659cac-4c32-4a94-90aa-8be0c6be33b2-kube-api-access-fcpzg\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:35.001073 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:35.001045 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7c4r\" (UniqueName: \"kubernetes.io/projected/57272665-8b2d-4854-bcb6-98f662716f4c-kube-api-access-s7c4r\") pod \"kube-state-metrics-5dc8d7d89b-ztrlc\" (UID: \"57272665-8b2d-4854-bcb6-98f662716f4c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc" Nov 04 03:17:35.166456 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:35.166427 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jf99x" Nov 04 03:17:35.189467 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:35.189434 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p8x4k" event={"ID":"c63fa297-f5dd-4c87-aa7b-f691ad61cb45","Type":"ContainerStarted","Data":"cd4f682f9bfa185730033d8094c13b7cfb773e02b818c82dd78783f6a00bc8cd"} Nov 04 03:17:35.194238 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:35.191648 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jg2pw" event={"ID":"9acbca43-6baa-4f8f-9eee-eb08dc731588","Type":"ContainerStarted","Data":"adfc5f7f686dafa6b0d02fd4415d7ff7fa58cea63e0225c2885ea6016dae40c4"} Nov 04 03:17:35.268130 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:35.268101 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc" Nov 04 03:17:35.289207 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:35.289179 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:35.599286 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:35.599195 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-jf99x"] Nov 04 03:17:35.642640 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:17:35.642585 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda040e5d8_7316_4a37_a045_9eb7c2eaf607.slice/crio-2b9d93ac2db03e8dfc74e8608d2ceaeec92c843309e60201ee4057bc5fa939bb WatchSource:0}: Error finding container 2b9d93ac2db03e8dfc74e8608d2ceaeec92c843309e60201ee4057bc5fa939bb: Status 404 returned error can't find the container with id 2b9d93ac2db03e8dfc74e8608d2ceaeec92c843309e60201ee4057bc5fa939bb Nov 04 03:17:35.654414 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:35.654390 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc"] Nov 04 03:17:35.659441 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:17:35.659404 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57272665_8b2d_4854_bcb6_98f662716f4c.slice/crio-f99e820db2e951c4e4a712acb3cd6c6d622e86bc03be5ed241a895818fb6e00c WatchSource:0}: Error finding container f99e820db2e951c4e4a712acb3cd6c6d622e86bc03be5ed241a895818fb6e00c: Status 404 returned error can't find the container with id f99e820db2e951c4e4a712acb3cd6c6d622e86bc03be5ed241a895818fb6e00c Nov 04 03:17:36.197770 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:36.197744 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jf99x" event={"ID":"a040e5d8-7316-4a37-a045-9eb7c2eaf607","Type":"ContainerStarted","Data":"6dc035430122bd10dfedc4f61a53172c6c64415dfcae70bf6c3fc3abcbb8dca7"} Nov 04 03:17:36.197870 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:36.197799 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jf99x" event={"ID":"a040e5d8-7316-4a37-a045-9eb7c2eaf607","Type":"ContainerStarted","Data":"2c216cc6f224e5bae0ac599de81568a82769fbb4617fa123631002f3fb03076e"} Nov 04 03:17:36.197870 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:36.197827 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jf99x" event={"ID":"a040e5d8-7316-4a37-a045-9eb7c2eaf607","Type":"ContainerStarted","Data":"2b9d93ac2db03e8dfc74e8608d2ceaeec92c843309e60201ee4057bc5fa939bb"} Nov 04 03:17:36.199533 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:36.199503 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jg2pw" event={"ID":"9acbca43-6baa-4f8f-9eee-eb08dc731588","Type":"ContainerStarted","Data":"fb3d6dc172a567bb02f964f3c429ae092c12c39d40af486ca1b4959e7d346f46"} Nov 04 03:17:36.199649 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:36.199608 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jg2pw" Nov 04 03:17:36.200944 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:36.200922 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wv6zh" event={"ID":"44f960f8-1c9c-4a1b-a422-5477a6d49316","Type":"ContainerStarted","Data":"32ef26ff41261931a9ce3245acf92bf3b7fac4925d973681ea39d331cb04e3db"} Nov 04 03:17:36.202115 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:36.202098 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc" event={"ID":"57272665-8b2d-4854-bcb6-98f662716f4c","Type":"ContainerStarted","Data":"f99e820db2e951c4e4a712acb3cd6c6d622e86bc03be5ed241a895818fb6e00c"} Nov 04 03:17:36.203145 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:36.203125 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7c9tl" event={"ID":"16659cac-4c32-4a94-90aa-8be0c6be33b2","Type":"ContainerStarted","Data":"5661b48df391450ec18633edf94716bde74a07933d0ef030521ad754a4909bf6"} Nov 04 03:17:36.317440 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:36.317380 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-jg2pw" podStartSLOduration=2.955954537 podStartE2EDuration="5.317366312s" totalImagesPullingTime="2.361411775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:17:36.316804099 +0000 UTC m=+53.914351912" watchObservedRunningTime="2026-11-04 03:17:36.317366312 +0000 UTC m=+53.914914105" Nov 04 03:17:36.855274 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:36.855204 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wv6zh" podStartSLOduration=2.556187392 podStartE2EDuration="5.85518371s" totalImagesPullingTime="3.298996318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:17:36.434065144 +0000 UTC m=+54.031612941" watchObservedRunningTime="2026-11-04 03:17:36.85518371 +0000 UTC m=+54.452731508" Nov 04 03:17:36.855710 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:36.855651 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 04 03:17:36.857944 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:36.857918 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:36.961512 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:36.961431 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 04 03:17:36.961754 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:36.961730 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 04 03:17:36.961830 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:36.961730 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 04 03:17:36.961830 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:36.961811 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-mwn26\"" Nov 04 03:17:36.961893 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:36.961835 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 04 03:17:36.961930 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:36.961894 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 04 03:17:36.961976 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:36.961940 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 04 03:17:36.961976 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:36.961955 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 04 03:17:36.961976 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:36.961896 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 04 03:17:36.962106 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:36.962005 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 04 03:17:36.968970 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:36.968942 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 04 03:17:37.032129 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.032091 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-config-out\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.032322 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.032143 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.032322 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.032174 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.032322 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.032241 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.032322 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.032300 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-web-config\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.032537 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.032332 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.032537 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.032361 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.032537 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.032392 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.032537 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.032417 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f99pm\" (UniqueName: \"kubernetes.io/projected/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-kube-api-access-f99pm\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.032537 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.032442 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-config-volume\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.032537 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.032464 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.032537 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.032493 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.032537 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.032518 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.133418 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.133379 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.133598 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.133465 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-web-config\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.133598 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.133502 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.133598 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.133579 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.133805 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.133613 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.133805 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.133657 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f99pm\" (UniqueName: \"kubernetes.io/projected/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-kube-api-access-f99pm\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.133805 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.133684 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-config-volume\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.133805 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.133705 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.133805 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.133735 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.133805 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.133762 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.133805 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.133800 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-config-out\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.134122 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.133826 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.134122 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.133851 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.136098 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.134748 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.136098 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.135340 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.136321 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.136269 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.137791 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.137268 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.137791 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.137729 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-config-volume\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.137986 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.137967 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-web-config\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.139110 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.139020 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-config-out\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.139247 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.139219 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.139444 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.139425 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.139843 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.139819 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.139954 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.139933 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.141089 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.141061 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.197415 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.197377 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f99pm\" (UniqueName: \"kubernetes.io/projected/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-kube-api-access-f99pm\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.208935 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.208894 2578 generic.go:356] "Generic (PLEG): container finished" podID="16659cac-4c32-4a94-90aa-8be0c6be33b2" containerID="7c17d08aee32ed9811d81e7bc47f2635f8bc415a63cf65ecafc82dc3f89f2367" exitCode=0 Nov 04 03:17:37.209090 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.209003 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7c9tl" event={"ID":"16659cac-4c32-4a94-90aa-8be0c6be33b2","Type":"ContainerDied","Data":"7c17d08aee32ed9811d81e7bc47f2635f8bc415a63cf65ecafc82dc3f89f2367"} Nov 04 03:17:37.213841 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.213172 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p8x4k" event={"ID":"c63fa297-f5dd-4c87-aa7b-f691ad61cb45","Type":"ContainerStarted","Data":"8e0b0cb3e3d604f5505c468bd24ce6d9e3a037c8adaffe96b8dffe24ca0da79b"} Nov 04 03:17:37.247105 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.245646 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7c4874f885-hqvmr"] Nov 04 03:17:37.248797 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.248769 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:37.301793 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.301764 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 04 03:17:37.301904 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.301859 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 04 03:17:37.302096 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.302066 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 04 03:17:37.303873 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.302206 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 04 03:17:37.303873 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.302269 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-124bnl54asoc8\"" Nov 04 03:17:37.303873 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.302346 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-tdnkm\"" Nov 04 03:17:37.303873 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.302380 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 04 03:17:37.313146 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.311471 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7c4874f885-hqvmr"] Nov 04 03:17:37.437259 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.437232 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d6ebb345-a666-4ece-9afe-c1be0a44867b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:37.437364 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.437271 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6ebb345-a666-4ece-9afe-c1be0a44867b-tmp\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:37.437423 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.437409 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d6ebb345-a666-4ece-9afe-c1be0a44867b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:37.437483 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.437442 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mr7j\" (UniqueName: \"kubernetes.io/projected/d6ebb345-a666-4ece-9afe-c1be0a44867b-kube-api-access-5mr7j\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:37.437539 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.437482 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d6ebb345-a666-4ece-9afe-c1be0a44867b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:37.437539 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.437514 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d6ebb345-a666-4ece-9afe-c1be0a44867b-secret-grpc-tls\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:37.437660 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.437554 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d6ebb345-a666-4ece-9afe-c1be0a44867b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:37.437660 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.437586 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d6ebb345-a666-4ece-9afe-c1be0a44867b-secret-thanos-querier-tls\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:37.437770 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.437665 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6ebb345-a666-4ece-9afe-c1be0a44867b-metrics-client-ca\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:37.446844 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.446786 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-p8x4k" podStartSLOduration=3.062741795 podStartE2EDuration="6.446773238s" totalImagesPullingTime="3.384031443s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 03:17:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:17:37.445973963 +0000 UTC m=+55.043521782" watchObservedRunningTime="2026-11-04 03:17:37.446773238 +0000 UTC m=+55.044321032" Nov 04 03:17:37.478249 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.478228 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:37.538926 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.538893 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d6ebb345-a666-4ece-9afe-c1be0a44867b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:37.539044 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.538930 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6ebb345-a666-4ece-9afe-c1be0a44867b-tmp\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:37.539044 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.538977 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d6ebb345-a666-4ece-9afe-c1be0a44867b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:37.539044 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.539002 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mr7j\" (UniqueName: \"kubernetes.io/projected/d6ebb345-a666-4ece-9afe-c1be0a44867b-kube-api-access-5mr7j\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:37.539044 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.539037 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d6ebb345-a666-4ece-9afe-c1be0a44867b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:37.539249 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.539065 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d6ebb345-a666-4ece-9afe-c1be0a44867b-secret-grpc-tls\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:37.539249 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.539104 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d6ebb345-a666-4ece-9afe-c1be0a44867b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:37.539249 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.539137 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d6ebb345-a666-4ece-9afe-c1be0a44867b-secret-thanos-querier-tls\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:37.539249 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.539189 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6ebb345-a666-4ece-9afe-c1be0a44867b-metrics-client-ca\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:37.540194 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.540161 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6ebb345-a666-4ece-9afe-c1be0a44867b-metrics-client-ca\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:37.540503 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.540477 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6ebb345-a666-4ece-9afe-c1be0a44867b-tmp\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:37.542370 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.542343 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d6ebb345-a666-4ece-9afe-c1be0a44867b-secret-grpc-tls\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:37.542812 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.542765 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d6ebb345-a666-4ece-9afe-c1be0a44867b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:37.542812 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.542767 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d6ebb345-a666-4ece-9afe-c1be0a44867b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:37.543192 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.543172 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d6ebb345-a666-4ece-9afe-c1be0a44867b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:37.543607 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.543585 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d6ebb345-a666-4ece-9afe-c1be0a44867b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:37.544103 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.544085 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d6ebb345-a666-4ece-9afe-c1be0a44867b-secret-thanos-querier-tls\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:37.601792 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.601772 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mr7j\" (UniqueName: \"kubernetes.io/projected/d6ebb345-a666-4ece-9afe-c1be0a44867b-kube-api-access-5mr7j\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:37.875598 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.875561 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:37.909730 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:37.909701 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 04 03:17:37.912955 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:17:37.912923 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd10b8919_8b30_4ee5_b4f9_2ea7794bfd31.slice/crio-6132bb22558201eafc6fe42434f09c31763011a7226766084223d4783a699f8f WatchSource:0}: Error finding container 6132bb22558201eafc6fe42434f09c31763011a7226766084223d4783a699f8f: Status 404 returned error can't find the container with id 6132bb22558201eafc6fe42434f09c31763011a7226766084223d4783a699f8f Nov 04 03:17:38.219180 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:38.219081 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc" event={"ID":"57272665-8b2d-4854-bcb6-98f662716f4c","Type":"ContainerStarted","Data":"78fffd3783f87c1469796a57696cb5571351054000978bcf8f1956761bd4a48f"} Nov 04 03:17:38.219460 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:38.219438 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc" event={"ID":"57272665-8b2d-4854-bcb6-98f662716f4c","Type":"ContainerStarted","Data":"d418217355b7f7fbe015207ff781f6b894368ac2fa6d420af0ddcfde44aa441e"} Nov 04 03:17:38.219638 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:38.219606 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc" event={"ID":"57272665-8b2d-4854-bcb6-98f662716f4c","Type":"ContainerStarted","Data":"3e9175c1562d2ac42c5f9ff6fd65c2727215fd26cc5fc2023770333fc64f4dde"} Nov 04 03:17:38.221145 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:38.221118 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31","Type":"ContainerStarted","Data":"6132bb22558201eafc6fe42434f09c31763011a7226766084223d4783a699f8f"} Nov 04 03:17:38.223608 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:38.223584 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7c9tl" event={"ID":"16659cac-4c32-4a94-90aa-8be0c6be33b2","Type":"ContainerStarted","Data":"73a716bb6834d045c984f806cd72a1465047cceb50cba01e7eb874f5fc4cb1f9"} Nov 04 03:17:38.223713 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:38.223647 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7c9tl" event={"ID":"16659cac-4c32-4a94-90aa-8be0c6be33b2","Type":"ContainerStarted","Data":"4dbb3f3f766591938567448c19aba3dafb9bc65f333d0f841b39d0c74476746a"} Nov 04 03:17:38.225862 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:38.225840 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jf99x" event={"ID":"a040e5d8-7316-4a37-a045-9eb7c2eaf607","Type":"ContainerStarted","Data":"eb533e4e27a634cecb9ca2c74017bb45c04652b592969abdfeaecfdaed900f7c"} Nov 04 03:17:38.251453 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:38.251427 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7c4874f885-hqvmr"] Nov 04 03:17:38.254726 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:17:38.254698 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6ebb345_a666_4ece_9afe_c1be0a44867b.slice/crio-760c6e388680b0e6510ec7d509e014d206711baf48b8309f94bb523169a34ede WatchSource:0}: Error finding container 760c6e388680b0e6510ec7d509e014d206711baf48b8309f94bb523169a34ede: Status 404 returned error can't find the container with id 760c6e388680b0e6510ec7d509e014d206711baf48b8309f94bb523169a34ede Nov 04 03:17:38.338390 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:38.338335 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc" podStartSLOduration=2.514686546 podStartE2EDuration="4.338319465s" totalImagesPullingTime="1.823632919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:17:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:17:38.337439603 +0000 UTC m=+55.934987398" watchObservedRunningTime="2026-11-04 03:17:38.338319465 +0000 UTC m=+55.935867295" Nov 04 03:17:38.455539 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:38.455472 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jf99x" podStartSLOduration=2.79477056 podStartE2EDuration="4.455456907s" totalImagesPullingTime="1.660686347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:17:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:17:38.454985113 +0000 UTC m=+56.052532910" watchObservedRunningTime="2026-11-04 03:17:38.455456907 +0000 UTC m=+56.053004700" Nov 04 03:17:38.578153 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:38.578092 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7c9tl" podStartSLOduration=3.090341672 podStartE2EDuration="4.578077702s" totalImagesPullingTime="1.40349365s" totalInitContainerRuntime="84.24238ms" isStatefulPod=true podCreationTimestamp="2026-11-04 03:17:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:17:38.576938031 +0000 UTC m=+56.174485825" watchObservedRunningTime="2026-11-04 03:17:38.578077702 +0000 UTC m=+56.175625496" Nov 04 03:17:39.233047 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.233010 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" event={"ID":"d6ebb345-a666-4ece-9afe-c1be0a44867b","Type":"ContainerStarted","Data":"760c6e388680b0e6510ec7d509e014d206711baf48b8309f94bb523169a34ede"} Nov 04 03:17:39.385125 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.385094 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-kf7dm"] Nov 04 03:17:39.387634 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.387598 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kf7dm" Nov 04 03:17:39.439654 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.439604 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 04 03:17:39.439785 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.439652 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-c8m9w\"" Nov 04 03:17:39.446090 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.446065 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-kf7dm"] Nov 04 03:17:39.459611 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.459533 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cd03c8ce-341a-4afe-93a1-2280392dd4f8-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-kf7dm\" (UID: \"cd03c8ce-341a-4afe-93a1-2280392dd4f8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-kf7dm" Nov 04 03:17:39.459885 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.459774 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtnzk\" (UniqueName: \"kubernetes.io/projected/cd03c8ce-341a-4afe-93a1-2280392dd4f8-kube-api-access-gtnzk\") pod \"monitoring-plugin-77f4f76594-kf7dm\" (UID: \"cd03c8ce-341a-4afe-93a1-2280392dd4f8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-kf7dm" Nov 04 03:17:39.553319 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.553284 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-748ff79954-pr5lc"] Nov 04 03:17:39.555410 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.555390 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" Nov 04 03:17:39.561015 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.560994 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-secret-metrics-server-tls\") pod \"metrics-server-748ff79954-pr5lc\" (UID: \"bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271\") " pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" Nov 04 03:17:39.561127 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.561047 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-748ff79954-pr5lc\" (UID: \"bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271\") " pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" Nov 04 03:17:39.561209 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.561130 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-audit-log\") pod \"metrics-server-748ff79954-pr5lc\" (UID: \"bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271\") " pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" Nov 04 03:17:39.561209 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.561176 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cd03c8ce-341a-4afe-93a1-2280392dd4f8-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-kf7dm\" (UID: \"cd03c8ce-341a-4afe-93a1-2280392dd4f8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-kf7dm" Nov 04 03:17:39.561310 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.561223 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbcgk\" (UniqueName: \"kubernetes.io/projected/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-kube-api-access-tbcgk\") pod \"metrics-server-748ff79954-pr5lc\" (UID: \"bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271\") " pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" Nov 04 03:17:39.561310 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.561284 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-metrics-server-audit-profiles\") pod \"metrics-server-748ff79954-pr5lc\" (UID: \"bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271\") " pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" Nov 04 03:17:39.561412 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.561332 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtnzk\" (UniqueName: \"kubernetes.io/projected/cd03c8ce-341a-4afe-93a1-2280392dd4f8-kube-api-access-gtnzk\") pod \"monitoring-plugin-77f4f76594-kf7dm\" (UID: \"cd03c8ce-341a-4afe-93a1-2280392dd4f8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-kf7dm" Nov 04 03:17:39.561412 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.561375 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-secret-metrics-server-client-certs\") pod \"metrics-server-748ff79954-pr5lc\" (UID: \"bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271\") " pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" Nov 04 03:17:39.561508 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.561415 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-client-ca-bundle\") pod \"metrics-server-748ff79954-pr5lc\" (UID: \"bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271\") " pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" Nov 04 03:17:39.564264 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.564242 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cd03c8ce-341a-4afe-93a1-2280392dd4f8-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-kf7dm\" (UID: \"cd03c8ce-341a-4afe-93a1-2280392dd4f8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-kf7dm" Nov 04 03:17:39.607671 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.607644 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 04 03:17:39.610849 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.610824 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 04 03:17:39.610993 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.610975 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 04 03:17:39.611065 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.611047 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4fbdz\"" Nov 04 03:17:39.611185 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.611168 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5drsvv7kbdor6\"" Nov 04 03:17:39.615648 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.615603 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-748ff79954-pr5lc"] Nov 04 03:17:39.624048 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.624021 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtnzk\" (UniqueName: \"kubernetes.io/projected/cd03c8ce-341a-4afe-93a1-2280392dd4f8-kube-api-access-gtnzk\") pod \"monitoring-plugin-77f4f76594-kf7dm\" (UID: \"cd03c8ce-341a-4afe-93a1-2280392dd4f8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-kf7dm" Nov 04 03:17:39.662026 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.661994 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbcgk\" (UniqueName: \"kubernetes.io/projected/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-kube-api-access-tbcgk\") pod \"metrics-server-748ff79954-pr5lc\" (UID: \"bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271\") " pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" Nov 04 03:17:39.662140 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.662055 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-metrics-server-audit-profiles\") pod \"metrics-server-748ff79954-pr5lc\" (UID: \"bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271\") " pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" Nov 04 03:17:39.662140 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.662100 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-secret-metrics-server-client-certs\") pod \"metrics-server-748ff79954-pr5lc\" (UID: \"bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271\") " pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" Nov 04 03:17:39.662140 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.662129 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-client-ca-bundle\") pod \"metrics-server-748ff79954-pr5lc\" (UID: \"bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271\") " pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" Nov 04 03:17:39.662273 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.662168 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-secret-metrics-server-tls\") pod \"metrics-server-748ff79954-pr5lc\" (UID: \"bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271\") " pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" Nov 04 03:17:39.662273 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.662210 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-748ff79954-pr5lc\" (UID: \"bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271\") " pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" Nov 04 03:17:39.662273 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.662255 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-audit-log\") pod \"metrics-server-748ff79954-pr5lc\" (UID: \"bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271\") " pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" Nov 04 03:17:39.662860 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.662816 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-audit-log\") pod \"metrics-server-748ff79954-pr5lc\" (UID: \"bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271\") " pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" Nov 04 03:17:39.663293 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.663256 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-748ff79954-pr5lc\" (UID: \"bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271\") " pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" Nov 04 03:17:39.665567 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.665542 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-secret-metrics-server-client-certs\") pod \"metrics-server-748ff79954-pr5lc\" (UID: \"bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271\") " pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" Nov 04 03:17:39.666363 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.666346 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-client-ca-bundle\") pod \"metrics-server-748ff79954-pr5lc\" (UID: \"bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271\") " pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" Nov 04 03:17:39.666578 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.666555 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-secret-metrics-server-tls\") pod \"metrics-server-748ff79954-pr5lc\" (UID: \"bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271\") " pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" Nov 04 03:17:39.708288 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.708251 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 04 03:17:39.714142 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.714114 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-metrics-server-audit-profiles\") pod \"metrics-server-748ff79954-pr5lc\" (UID: \"bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271\") " pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" Nov 04 03:17:39.716597 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.716573 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kf7dm" Nov 04 03:17:39.723464 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.723446 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbcgk\" (UniqueName: \"kubernetes.io/projected/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-kube-api-access-tbcgk\") pod \"metrics-server-748ff79954-pr5lc\" (UID: \"bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271\") " pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" Nov 04 03:17:39.875821 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:39.875742 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" Nov 04 03:17:40.094173 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.094021 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-kf7dm"] Nov 04 03:17:40.096740 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:17:40.096712 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd03c8ce_341a_4afe_93a1_2280392dd4f8.slice/crio-1f357050039df0c66f46c3927903c9f913ef0340757c5043ead5991a62a0963b WatchSource:0}: Error finding container 1f357050039df0c66f46c3927903c9f913ef0340757c5043ead5991a62a0963b: Status 404 returned error can't find the container with id 1f357050039df0c66f46c3927903c9f913ef0340757c5043ead5991a62a0963b Nov 04 03:17:40.238827 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.238738 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kf7dm" event={"ID":"cd03c8ce-341a-4afe-93a1-2280392dd4f8","Type":"ContainerStarted","Data":"1f357050039df0c66f46c3927903c9f913ef0340757c5043ead5991a62a0963b"} Nov 04 03:17:40.240671 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.240639 2578 generic.go:356] "Generic (PLEG): container finished" podID="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" containerID="f4ac97bd1d0e01b9f860c3ffeb1a6073e2aa15efea50981c2053c645efdbe607" exitCode=0 Nov 04 03:17:40.240811 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.240677 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31","Type":"ContainerDied","Data":"f4ac97bd1d0e01b9f860c3ffeb1a6073e2aa15efea50981c2053c645efdbe607"} Nov 04 03:17:40.248768 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.248745 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-748ff79954-pr5lc"] Nov 04 03:17:40.251957 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:17:40.251933 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf7bd2f_e4c8_469f_ad1b_ffe5cf1fc271.slice/crio-ea615393cda142a2d94cdcc77c191ac97a95d132f96c4c9a6636513c52ea39f3 WatchSource:0}: Error finding container ea615393cda142a2d94cdcc77c191ac97a95d132f96c4c9a6636513c52ea39f3: Status 404 returned error can't find the container with id ea615393cda142a2d94cdcc77c191ac97a95d132f96c4c9a6636513c52ea39f3 Nov 04 03:17:40.337594 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.337562 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-745f9d6695-rlnsq"] Nov 04 03:17:40.340301 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.340276 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:40.369117 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.369076 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ab83526-361f-41de-b480-79fc45d7373a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:40.369250 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.369167 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4ab83526-361f-41de-b480-79fc45d7373a-federate-client-tls\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:40.369250 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.369245 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94vqb\" (UniqueName: \"kubernetes.io/projected/4ab83526-361f-41de-b480-79fc45d7373a-kube-api-access-94vqb\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:40.369376 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.369285 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ab83526-361f-41de-b480-79fc45d7373a-serving-certs-ca-bundle\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:40.369376 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.369310 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4ab83526-361f-41de-b480-79fc45d7373a-secret-telemeter-client\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:40.369376 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.369367 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4ab83526-361f-41de-b480-79fc45d7373a-telemeter-client-tls\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:40.369535 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.369400 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ab83526-361f-41de-b480-79fc45d7373a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:40.369535 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.369438 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ab83526-361f-41de-b480-79fc45d7373a-metrics-client-ca\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:40.392828 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.392803 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 04 03:17:40.393078 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.393057 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 04 03:17:40.393190 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.393126 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 04 03:17:40.393190 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.393125 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-sn5zj\"" Nov 04 03:17:40.394938 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.394917 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 04 03:17:40.394938 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.394935 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 04 03:17:40.422572 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.422546 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 04 03:17:40.444275 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.444245 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-745f9d6695-rlnsq"] Nov 04 03:17:40.469861 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.469835 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ab83526-361f-41de-b480-79fc45d7373a-serving-certs-ca-bundle\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:40.469959 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.469869 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4ab83526-361f-41de-b480-79fc45d7373a-secret-telemeter-client\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:40.469959 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.469924 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4ab83526-361f-41de-b480-79fc45d7373a-telemeter-client-tls\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:40.470084 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.469955 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ab83526-361f-41de-b480-79fc45d7373a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:40.470084 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.469992 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ab83526-361f-41de-b480-79fc45d7373a-metrics-client-ca\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:40.470084 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.470052 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ab83526-361f-41de-b480-79fc45d7373a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:40.470225 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.470096 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4ab83526-361f-41de-b480-79fc45d7373a-federate-client-tls\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:40.470225 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.470167 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94vqb\" (UniqueName: \"kubernetes.io/projected/4ab83526-361f-41de-b480-79fc45d7373a-kube-api-access-94vqb\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:40.470588 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.470562 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ab83526-361f-41de-b480-79fc45d7373a-serving-certs-ca-bundle\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:40.470869 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.470817 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ab83526-361f-41de-b480-79fc45d7373a-metrics-client-ca\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:40.470954 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.470907 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ab83526-361f-41de-b480-79fc45d7373a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:40.473375 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.473345 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4ab83526-361f-41de-b480-79fc45d7373a-telemeter-client-tls\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:40.473375 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.473364 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ab83526-361f-41de-b480-79fc45d7373a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:40.473539 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.473524 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4ab83526-361f-41de-b480-79fc45d7373a-federate-client-tls\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:40.473591 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.473538 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4ab83526-361f-41de-b480-79fc45d7373a-secret-telemeter-client\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:40.533715 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.533685 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94vqb\" (UniqueName: \"kubernetes.io/projected/4ab83526-361f-41de-b480-79fc45d7373a-kube-api-access-94vqb\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:40.653100 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:40.653066 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:41.084737 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:41.084649 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-745f9d6695-rlnsq"] Nov 04 03:17:41.087387 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:17:41.087354 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ab83526_361f_41de_b480_79fc45d7373a.slice/crio-c4998297ab19ab972cd9dbc2637cfe3b110924f917064937d89ed0ffc0959fbd WatchSource:0}: Error finding container c4998297ab19ab972cd9dbc2637cfe3b110924f917064937d89ed0ffc0959fbd: Status 404 returned error can't find the container with id c4998297ab19ab972cd9dbc2637cfe3b110924f917064937d89ed0ffc0959fbd Nov 04 03:17:41.254825 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:41.254790 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" event={"ID":"d6ebb345-a666-4ece-9afe-c1be0a44867b","Type":"ContainerStarted","Data":"38b1179a0f01d8ce265ee5f3acd77e497d0655064dcd73870346cb6ad7fb4066"} Nov 04 03:17:41.255230 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:41.254901 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" event={"ID":"d6ebb345-a666-4ece-9afe-c1be0a44867b","Type":"ContainerStarted","Data":"e82b9270479c5b347ce270c6d7668dcb412eb0c545d478f71b35d7cda960fdef"} Nov 04 03:17:41.255230 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:41.254945 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" event={"ID":"d6ebb345-a666-4ece-9afe-c1be0a44867b","Type":"ContainerStarted","Data":"1557b9165f6540c7fe0505723ed7b8c7a5fbb895f18648cfa2e79051362cbb74"} Nov 04 03:17:41.257265 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:41.257200 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" event={"ID":"4ab83526-361f-41de-b480-79fc45d7373a","Type":"ContainerStarted","Data":"c4998297ab19ab972cd9dbc2637cfe3b110924f917064937d89ed0ffc0959fbd"} Nov 04 03:17:41.258632 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:41.258588 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" event={"ID":"bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271","Type":"ContainerStarted","Data":"ea615393cda142a2d94cdcc77c191ac97a95d132f96c4c9a6636513c52ea39f3"} Nov 04 03:17:41.475524 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:41.475449 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:43.268391 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.268348 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" event={"ID":"bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271","Type":"ContainerStarted","Data":"1b5aa405f00b1dd45ee5762aae61982c22d76aea95abe62d85672337e70c5b79"} Nov 04 03:17:43.272554 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.272525 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31","Type":"ContainerStarted","Data":"c9a65ccf157d7695f25dd196151bee5ecfbb9b0ad205c85e7060d0537fd40706"} Nov 04 03:17:43.272701 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.272663 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31","Type":"ContainerStarted","Data":"cb5b83e3d9d660b2441013aee8be5c36b6faed092a60491930b9de1a974dd478"} Nov 04 03:17:43.272758 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.272708 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31","Type":"ContainerStarted","Data":"39c4768b174ad259edc7e17463fadf1eee93694e930100e24c578bd0f207db44"} Nov 04 03:17:43.272758 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.272747 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31","Type":"ContainerStarted","Data":"b1d85abe163b8f70f8552f5dc26ba32f89375f6af2c6043ae5f5ef1c71573718"} Nov 04 03:17:43.272867 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.272785 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31","Type":"ContainerStarted","Data":"a3be12ddcd9acb0895e6f0a5b93c51bf1431a92793e1cbc155c89fbb163ff0b0"} Nov 04 03:17:43.272867 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.272824 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31","Type":"ContainerStarted","Data":"8254ce7d4c3adaa19ae449cdd41cf2b74f62812a3853535de6ade8f79389fa13"} Nov 04 03:17:43.276237 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.276207 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" event={"ID":"d6ebb345-a666-4ece-9afe-c1be0a44867b","Type":"ContainerStarted","Data":"711270376d9322e2b210a17bdb9ecc6c523f19925d728a5cb9b89f737685178d"} Nov 04 03:17:43.276340 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.276279 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" event={"ID":"d6ebb345-a666-4ece-9afe-c1be0a44867b","Type":"ContainerStarted","Data":"7f73c4dd6afa31ab503d6f4f18471d86eddafc781a508342c2c9e1fdd65c7b65"} Nov 04 03:17:43.276340 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.276326 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" event={"ID":"d6ebb345-a666-4ece-9afe-c1be0a44867b","Type":"ContainerStarted","Data":"09c038cda4862c27bfc4a177dded9659f7f1aecc6642cbc15f23b352af306de0"} Nov 04 03:17:43.276466 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.276366 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:43.278026 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.277999 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kf7dm" event={"ID":"cd03c8ce-341a-4afe-93a1-2280392dd4f8","Type":"ContainerStarted","Data":"7ea36278108c62d0c783c54f4950c0a4c4e48e60fa62d0696b6fe6e929cc9a62"} Nov 04 03:17:43.278137 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.278112 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kf7dm" Nov 04 03:17:43.283796 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.283765 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kf7dm" Nov 04 03:17:43.386417 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.386359 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" podStartSLOduration=2.039147896 podStartE2EDuration="4.386339849s" totalImagesPullingTime="2.347191953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:17:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:17:43.385580869 +0000 UTC m=+60.983128664" watchObservedRunningTime="2026-11-04 03:17:43.386339849 +0000 UTC m=+60.983887713" Nov 04 03:17:43.515746 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.515698 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.9168326110000002 podStartE2EDuration="7.515683681s" totalImagesPullingTime="3.583965679s" totalInitContainerRuntime="14.885391ms" isStatefulPod=false podCreationTimestamp="2026-11-04 03:17:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:17:43.514863428 +0000 UTC m=+61.112411247" watchObservedRunningTime="2026-11-04 03:17:43.515683681 +0000 UTC m=+61.113231476" Nov 04 03:17:43.640827 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.640760 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" podStartSLOduration=2.827790611 podStartE2EDuration="6.640742105s" totalImagesPullingTime="3.812951494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:17:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:17:43.63979654 +0000 UTC m=+61.237344368" watchObservedRunningTime="2026-11-04 03:17:43.640742105 +0000 UTC m=+61.238289899" Nov 04 03:17:43.713698 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.713591 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 03:17:43.716776 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.716754 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.756292 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.756234 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kf7dm" podStartSLOduration=2.500387183 podStartE2EDuration="4.756219649s" totalImagesPullingTime="2.255832466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:17:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:17:43.755176635 +0000 UTC m=+61.352724444" watchObservedRunningTime="2026-11-04 03:17:43.756219649 +0000 UTC m=+61.353767443" Nov 04 03:17:43.769936 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.769906 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 04 03:17:43.770079 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.769958 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 04 03:17:43.770079 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.769975 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 04 03:17:43.770677 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.770656 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 04 03:17:43.801339 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.801312 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.801480 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.801342 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.801480 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.801362 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.801480 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.801378 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.801611 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.801494 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-config\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.801611 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.801544 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-web-config\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.801611 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.801567 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-config-out\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.801830 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.801681 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.801830 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.801715 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.801830 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.801740 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rmmf\" (UniqueName: \"kubernetes.io/projected/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-kube-api-access-9rmmf\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.801830 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.801798 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.801996 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.801866 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.801996 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.801885 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.801996 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.801947 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.801996 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.801983 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.802165 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.802034 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.802165 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.802068 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.802165 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.802099 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.802258 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.802162 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.802258 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.802191 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.819959 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.819930 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 04 03:17:43.820065 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.819958 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 04 03:17:43.822259 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.822237 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 03:17:43.869852 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.869816 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 04 03:17:43.869852 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.869840 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 04 03:17:43.869852 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.869816 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 04 03:17:43.870117 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.869844 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 04 03:17:43.870117 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.869820 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 04 03:17:43.870117 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.869896 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e4spppbfh1spp\"" Nov 04 03:17:43.870117 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.869998 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-bh6gw\"" Nov 04 03:17:43.870117 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.870083 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 04 03:17:43.870270 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.870141 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 04 03:17:43.871380 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.871359 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 04 03:17:43.903313 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.903286 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.903313 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.903315 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.903479 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.903332 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.903479 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.903351 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.903479 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.903383 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-config\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.903479 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.903412 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-web-config\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.903479 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.903441 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-config-out\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.903479 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.903471 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.903796 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.903496 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.903796 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.903519 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rmmf\" (UniqueName: \"kubernetes.io/projected/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-kube-api-access-9rmmf\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.903796 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.903541 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.903796 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.903583 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.903796 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.903611 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.903796 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.903690 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.903796 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.903717 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.903796 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.903762 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.903796 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.903795 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.904563 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.903825 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.904563 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.903863 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.904563 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.903908 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.904563 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.904537 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.905358 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.905332 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.906751 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.906721 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.906915 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.906895 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.907013 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.906958 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.907166 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.907147 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-web-config\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.907337 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.907319 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.907469 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.907446 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.907544 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.907473 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.908028 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.907875 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.908134 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.908108 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-config-out\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.908594 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.908552 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.908815 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.908792 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.908960 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.908939 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-config\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.909652 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.909601 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.909933 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.909908 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.909933 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.909925 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.910445 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.910423 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.910497 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.910467 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:43.964866 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:43.964793 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rmmf\" (UniqueName: \"kubernetes.io/projected/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-kube-api-access-9rmmf\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:44.036584 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:44.036547 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:44.284246 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:44.284200 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" event={"ID":"4ab83526-361f-41de-b480-79fc45d7373a","Type":"ContainerStarted","Data":"42ceff03fa360fe9b3c0ac4ad4744b327c2409f6d5b36438043fb38c76edd439"} Nov 04 03:17:44.284728 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:44.284267 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" event={"ID":"4ab83526-361f-41de-b480-79fc45d7373a","Type":"ContainerStarted","Data":"d7820f1dd5b6a0caf08fbf9a11e21e879186d1cbe91b29c5a44d1d50dd14c155"} Nov 04 03:17:44.284728 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:44.284286 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" event={"ID":"4ab83526-361f-41de-b480-79fc45d7373a","Type":"ContainerStarted","Data":"15d6305110a4181083ffb6379f432e6cf4f94ffb5d68cebc79becbb939225ff8"} Nov 04 03:17:44.407571 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:44.407514 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" podStartSLOduration=2.04843315 podStartE2EDuration="4.40749962s" totalImagesPullingTime="2.35906647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:17:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:17:44.406989378 +0000 UTC m=+62.004537171" watchObservedRunningTime="2026-11-04 03:17:44.40749962 +0000 UTC m=+62.005047414" Nov 04 03:17:44.470371 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:44.469985 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 03:17:44.472811 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:17:44.472772 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9c2dd4c_0b6e_4560_9b46_c62ee49cf49e.slice/crio-791796dc50b8bec54529e842614d63f8768af87659c325aa9ada29014e330c61 WatchSource:0}: Error finding container 791796dc50b8bec54529e842614d63f8768af87659c325aa9ada29014e330c61: Status 404 returned error can't find the container with id 791796dc50b8bec54529e842614d63f8768af87659c325aa9ada29014e330c61 Nov 04 03:17:45.289794 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:45.289762 2578 generic.go:356] "Generic (PLEG): container finished" podID="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" containerID="c5b0e0618d7907ea3905d068f6441be77893e0bdeb245dd8dab7618ad3fc1818" exitCode=0 Nov 04 03:17:45.290175 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:45.289910 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e","Type":"ContainerDied","Data":"c5b0e0618d7907ea3905d068f6441be77893e0bdeb245dd8dab7618ad3fc1818"} Nov 04 03:17:45.290175 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:45.289973 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e","Type":"ContainerStarted","Data":"791796dc50b8bec54529e842614d63f8768af87659c325aa9ada29014e330c61"} Nov 04 03:17:46.216319 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:46.216286 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jg2pw" Nov 04 03:17:47.946811 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:47.946773 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aa3f04c-b684-4426-8d46-a36d8c4e77be-metrics-certs\") pod \"network-metrics-daemon-dzg2c\" (UID: \"1aa3f04c-b684-4426-8d46-a36d8c4e77be\") " pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:17:47.947257 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:47.946844 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/834b5587-1820-4089-a925-ab992fe654b4-original-pull-secret\") pod \"global-pull-secret-syncer-8rmjv\" (UID: \"834b5587-1820-4089-a925-ab992fe654b4\") " pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:17:47.998593 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:47.998550 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 04 03:17:47.998593 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:47.998574 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 04 03:17:48.010113 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:48.010087 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aa3f04c-b684-4426-8d46-a36d8c4e77be-metrics-certs\") pod \"network-metrics-daemon-dzg2c\" (UID: \"1aa3f04c-b684-4426-8d46-a36d8c4e77be\") " pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:17:48.010291 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:48.010271 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/834b5587-1820-4089-a925-ab992fe654b4-original-pull-secret\") pod \"global-pull-secret-syncer-8rmjv\" (UID: \"834b5587-1820-4089-a925-ab992fe654b4\") " pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:17:48.047753 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:48.047712 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7xg7\" (UniqueName: \"kubernetes.io/projected/719911b7-0c11-4c79-bb86-fdc043d9f9ab-kube-api-access-n7xg7\") pod \"network-check-target-2hjpn\" (UID: \"719911b7-0c11-4c79-bb86-fdc043d9f9ab\") " pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:17:48.099234 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:48.099201 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 04 03:17:48.159536 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:48.159498 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 04 03:17:48.171348 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:48.171318 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7xg7\" (UniqueName: \"kubernetes.io/projected/719911b7-0c11-4c79-bb86-fdc043d9f9ab-kube-api-access-n7xg7\") pod \"network-check-target-2hjpn\" (UID: \"719911b7-0c11-4c79-bb86-fdc043d9f9ab\") " pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:17:48.327376 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:48.327345 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x9qdq\"" Nov 04 03:17:48.336024 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:48.335996 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:17:48.344940 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:48.344819 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-f24jt\"" Nov 04 03:17:48.347594 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:48.347557 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-kfskp\"" Nov 04 03:17:48.353175 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:48.353151 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:17:48.356594 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:48.356401 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:17:48.717338 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:48.717263 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dzg2c"] Nov 04 03:17:48.755080 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:17:48.755048 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aa3f04c_b684_4426_8d46_a36d8c4e77be.slice/crio-63559e597407dae564a620e85c94cbf6318bd09f5c75313d0e6fdd9fba2f126a WatchSource:0}: Error finding container 63559e597407dae564a620e85c94cbf6318bd09f5c75313d0e6fdd9fba2f126a: Status 404 returned error can't find the container with id 63559e597407dae564a620e85c94cbf6318bd09f5c75313d0e6fdd9fba2f126a Nov 04 03:17:48.766212 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:48.766183 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8rmjv"] Nov 04 03:17:48.768485 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:48.768462 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2hjpn"] Nov 04 03:17:48.771918 ip-10-0-133-6 kubenswrapper[2578]: W1104 03:17:48.771828 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod719911b7_0c11_4c79_bb86_fdc043d9f9ab.slice/crio-1da1db8c80e0d30a8d1ff45a275e72414ff80b95ee8e6dba298e60a9807c7740 WatchSource:0}: Error finding container 1da1db8c80e0d30a8d1ff45a275e72414ff80b95ee8e6dba298e60a9807c7740: Status 404 returned error can't find the container with id 1da1db8c80e0d30a8d1ff45a275e72414ff80b95ee8e6dba298e60a9807c7740 Nov 04 03:17:49.291465 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:49.291438 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:49.306522 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:49.306492 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dzg2c" event={"ID":"1aa3f04c-b684-4426-8d46-a36d8c4e77be","Type":"ContainerStarted","Data":"63559e597407dae564a620e85c94cbf6318bd09f5c75313d0e6fdd9fba2f126a"} Nov 04 03:17:49.310102 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:49.310079 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e","Type":"ContainerStarted","Data":"4923b65f9e3377eb7546b564bc8470d59713a6761433c1ccd7fbc667e707250c"} Nov 04 03:17:49.310234 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:49.310141 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e","Type":"ContainerStarted","Data":"dedb1c9816ecad494cbc4bde07bdea706f4a7aaffce47e5e101a9db1a532b062"} Nov 04 03:17:49.310234 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:49.310169 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e","Type":"ContainerStarted","Data":"2f40b0a79469bfb66ca1723dfed6cce23aab6b2e980135a50e11335832775639"} Nov 04 03:17:49.310234 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:49.310210 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e","Type":"ContainerStarted","Data":"8df5edc17acbce219b8ee1e04527c0a07bfa8e3ef8772a07acb7c46bbc69b457"} Nov 04 03:17:49.310354 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:49.310236 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e","Type":"ContainerStarted","Data":"5aea6e059cb33894f5559432ed56c5f1f503907e0955d2dfdc7419d997f5dbf8"} Nov 04 03:17:49.310354 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:49.310258 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e","Type":"ContainerStarted","Data":"d5918d555e4e664d0b9dc19f3ea84d5d728df2cb00e5dad2143ea52b72766645"} Nov 04 03:17:49.311341 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:49.311322 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2hjpn" event={"ID":"719911b7-0c11-4c79-bb86-fdc043d9f9ab","Type":"ContainerStarted","Data":"1da1db8c80e0d30a8d1ff45a275e72414ff80b95ee8e6dba298e60a9807c7740"} Nov 04 03:17:49.312411 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:49.312392 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8rmjv" event={"ID":"834b5587-1820-4089-a925-ab992fe654b4","Type":"ContainerStarted","Data":"75a2b016aaf3f6408b28d110698ecd053b2ae2ecbaa8aa7bcb68e1e6c2c30068"} Nov 04 03:17:49.559578 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:49.559469 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.00094009 podStartE2EDuration="6.559452038s" totalImagesPullingTime="3.541663721s" totalInitContainerRuntime="16.848227ms" isStatefulPod=false podCreationTimestamp="2026-11-04 03:17:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:17:49.557365746 +0000 UTC m=+67.154913541" watchObservedRunningTime="2026-11-04 03:17:49.559452038 +0000 UTC m=+67.156999831" Nov 04 03:17:51.322694 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:51.322656 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dzg2c" event={"ID":"1aa3f04c-b684-4426-8d46-a36d8c4e77be","Type":"ContainerStarted","Data":"7296bf1a0952ebd41c3815f62bce0f23fb143604a99c0fdecca4b02d9159f2a9"} Nov 04 03:17:51.323141 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:51.322718 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dzg2c" event={"ID":"1aa3f04c-b684-4426-8d46-a36d8c4e77be","Type":"ContainerStarted","Data":"ff7131e03aa6285c39abd8a9a9578100d124ca3856e9dfda0fa1fee4aeddb97f"} Nov 04 03:17:51.441086 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:51.441024 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dzg2c" podStartSLOduration=66.765556723 podStartE2EDuration="1m8.441007057s" totalImagesPullingTime="1.675450334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:16:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:17:51.440084933 +0000 UTC m=+69.037632727" watchObservedRunningTime="2026-11-04 03:17:51.441007057 +0000 UTC m=+69.038554863" Nov 04 03:17:53.332649 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:53.332589 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2hjpn" event={"ID":"719911b7-0c11-4c79-bb86-fdc043d9f9ab","Type":"ContainerStarted","Data":"ae5d1b045fce318f06005c422a3ed29acd80525214959b34389ec5ec28a0083f"} Nov 04 03:17:53.333129 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:53.332764 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:17:53.449221 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:53.449159 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2hjpn" podStartSLOduration=66.467078695 podStartE2EDuration="1m10.449143177s" totalImagesPullingTime="3.982064482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:16:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:17:53.447329675 +0000 UTC m=+71.044877471" watchObservedRunningTime="2026-11-04 03:17:53.449143177 +0000 UTC m=+71.046690970" Nov 04 03:17:54.037260 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:54.037207 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:54.665324 ip-10-0-133-6 kubenswrapper[2578]: I1104 03:17:54.665203 2578 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 03:17:54.665367 ip-10-0-133-6 systemd[1]: Stopping Kubernetes Kubelet... Nov 04 03:17:54.676069 ip-10-0-133-6 systemd[1]: kubelet.service: Deactivated successfully. Nov 04 03:17:54.676244 ip-10-0-133-6 systemd[1]: Stopped Kubernetes Kubelet. Nov 04 03:17:54.676290 ip-10-0-133-6 systemd[1]: kubelet.service: Consumed 4.769s CPU time. Nov 04 03:17:54.690541 ip-10-0-133-6 systemd[1]: Starting Kubernetes Kubelet... Nov 04 03:17:54.758369 ip-10-0-133-6 kubenswrapper[5953]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 03:17:54.758369 ip-10-0-133-6 kubenswrapper[5953]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 04 03:17:54.758369 ip-10-0-133-6 kubenswrapper[5953]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 03:17:54.758369 ip-10-0-133-6 kubenswrapper[5953]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 03:17:54.760746 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760661 5953 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:17:54.760746 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760744 5953 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:17:54.760746 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760748 5953 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:17:54.760746 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760751 5953 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:17:54.760746 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760754 5953 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:17:54.760973 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760757 5953 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:17:54.760973 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760764 5953 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:17:54.760973 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760767 5953 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:17:54.760973 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760770 5953 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:17:54.760973 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760773 5953 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:17:54.760973 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760775 5953 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:17:54.760973 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760778 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:17:54.760973 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760780 5953 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:17:54.760973 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760783 5953 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:17:54.760973 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760786 5953 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:17:54.760973 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760788 5953 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:17:54.760973 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760791 5953 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:17:54.760973 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760793 5953 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:17:54.760973 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760796 5953 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:17:54.760973 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760799 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:17:54.760973 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760801 5953 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:17:54.760973 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760805 5953 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:17:54.760973 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760807 5953 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:17:54.760973 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760810 5953 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:17:54.760973 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760813 5953 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:17:54.761511 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760815 5953 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:17:54.761511 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760818 5953 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:17:54.761511 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760821 5953 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:17:54.761511 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760824 5953 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:17:54.761511 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760826 5953 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:17:54.761511 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760829 5953 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:17:54.761511 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760832 5953 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:17:54.761511 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760835 5953 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:17:54.761511 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760837 5953 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:17:54.761511 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760839 5953 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:17:54.761511 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760842 5953 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:17:54.761511 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760844 5953 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:17:54.761511 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760847 5953 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:17:54.761511 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760849 5953 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:17:54.761511 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760853 5953 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:17:54.761511 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760855 5953 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:17:54.761511 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760858 5953 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:17:54.761511 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760860 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:17:54.761511 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760863 5953 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:17:54.761511 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760865 5953 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:17:54.762065 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760868 5953 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:17:54.762065 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760870 5953 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:17:54.762065 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760873 5953 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:17:54.762065 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760876 5953 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:17:54.762065 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760878 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:17:54.762065 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760881 5953 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:17:54.762065 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760883 5953 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:17:54.762065 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760886 5953 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:17:54.762065 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760889 5953 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:17:54.762065 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760892 5953 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:17:54.762065 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760894 5953 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:17:54.762065 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760897 5953 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:17:54.762065 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760899 5953 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:17:54.762065 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760902 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:17:54.762065 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760905 5953 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:17:54.762065 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760910 5953 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:17:54.762065 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760913 5953 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:17:54.762065 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760916 5953 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:17:54.762065 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760919 5953 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:17:54.762065 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760922 5953 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:17:54.762610 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760924 5953 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:17:54.762610 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760927 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:17:54.762610 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760929 5953 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:17:54.762610 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760932 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:17:54.762610 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760934 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:17:54.762610 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760937 5953 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:17:54.762610 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760939 5953 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:17:54.762610 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760942 5953 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:17:54.762610 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760945 5953 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:17:54.762610 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760948 5953 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:17:54.762610 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760950 5953 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:17:54.762610 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760953 5953 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:17:54.762610 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760956 5953 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:17:54.762610 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760958 5953 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:17:54.762610 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760961 5953 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:17:54.762610 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760963 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:17:54.762610 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760966 5953 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:17:54.762610 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760968 5953 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:17:54.762610 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760971 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:17:54.762610 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760975 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:17:54.763117 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760978 5953 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:17:54.763117 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760981 5953 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:17:54.763117 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760983 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:17:54.763117 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760988 5953 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:17:54.763117 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760993 5953 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:17:54.763117 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760997 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:17:54.763117 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.760999 5953 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:17:54.763117 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761004 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:17:54.763117 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761006 5953 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:17:54.763117 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761009 5953 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:17:54.763117 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761011 5953 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:17:54.763117 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761016 5953 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:17:54.763117 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761020 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:17:54.763117 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761024 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:17:54.763117 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761027 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:17:54.763117 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761029 5953 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:17:54.763117 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761032 5953 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:17:54.763117 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761034 5953 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:17:54.763596 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761037 5953 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:17:54.763596 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761039 5953 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:17:54.763596 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761042 5953 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:17:54.763596 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761834 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:17:54.763596 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761842 5953 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:17:54.763596 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761845 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:17:54.763596 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761848 5953 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:17:54.763596 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761851 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:17:54.763596 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761853 5953 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:17:54.763596 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761856 5953 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:17:54.763596 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761859 5953 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:17:54.763596 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761862 5953 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:17:54.763596 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761865 5953 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:17:54.763596 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761868 5953 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:17:54.763596 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761870 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:17:54.763596 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761873 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:17:54.763596 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761875 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:17:54.763596 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761878 5953 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:17:54.763596 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761880 5953 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:17:54.764066 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761883 5953 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:17:54.764066 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761886 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:17:54.764066 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761891 5953 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:17:54.764066 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761894 5953 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:17:54.764066 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761896 5953 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:17:54.764066 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761899 5953 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:17:54.764066 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761901 5953 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:17:54.764066 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761904 5953 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:17:54.764066 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761906 5953 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:17:54.764066 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761909 5953 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:17:54.764066 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761912 5953 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:17:54.764066 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761914 5953 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:17:54.764066 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761917 5953 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:17:54.764066 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761920 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:17:54.764066 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761922 5953 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:17:54.764066 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761925 5953 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:17:54.764066 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761928 5953 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:17:54.764066 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761932 5953 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:17:54.764066 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761934 5953 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:17:54.764066 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761938 5953 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:17:54.764535 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761940 5953 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:17:54.764535 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761943 5953 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:17:54.764535 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761945 5953 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:17:54.764535 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761948 5953 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:17:54.764535 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761951 5953 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:17:54.764535 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761953 5953 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:17:54.764535 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761956 5953 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:17:54.764535 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761958 5953 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:17:54.764535 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761961 5953 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:17:54.764535 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761964 5953 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:17:54.764535 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761966 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:17:54.764535 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761969 5953 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:17:54.764535 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761971 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:17:54.764535 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761976 5953 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:17:54.764535 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761981 5953 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:17:54.764535 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761984 5953 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:17:54.764535 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761986 5953 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:17:54.764535 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761989 5953 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:17:54.764535 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761992 5953 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:17:54.765008 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761994 5953 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:17:54.765008 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761997 5953 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:17:54.765008 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.761999 5953 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:17:54.765008 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762002 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:17:54.765008 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762005 5953 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:17:54.765008 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762007 5953 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:17:54.765008 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762010 5953 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:17:54.765008 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762013 5953 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:17:54.765008 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762015 5953 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:17:54.765008 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762019 5953 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:17:54.765008 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762024 5953 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:17:54.765008 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762029 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:17:54.765008 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762032 5953 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:17:54.765008 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762034 5953 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:17:54.765008 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762039 5953 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:17:54.765008 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762043 5953 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:17:54.765008 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762047 5953 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:17:54.765008 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762052 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:17:54.765008 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762055 5953 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:17:54.765008 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762060 5953 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:17:54.765547 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762063 5953 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:17:54.765547 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762068 5953 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:17:54.765547 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762072 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:17:54.765547 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762076 5953 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:17:54.765547 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762080 5953 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:17:54.765547 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762084 5953 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:17:54.765547 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762089 5953 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:17:54.765547 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762093 5953 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:17:54.765547 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762097 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:17:54.765547 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762101 5953 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:17:54.765547 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762104 5953 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:17:54.765547 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762109 5953 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:17:54.765547 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762112 5953 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:17:54.765547 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762116 5953 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:17:54.765547 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762120 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:17:54.765547 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762124 5953 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:17:54.765547 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762128 5953 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:17:54.765547 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762132 5953 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:17:54.765547 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762136 5953 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:17:54.766106 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762140 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:17:54.766106 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762145 5953 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:17:54.766106 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762150 5953 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:17:54.766106 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762154 5953 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:17:54.766106 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762159 5953 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:17:54.766106 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762163 5953 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:17:54.766106 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762167 5953 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:17:54.766106 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762171 5953 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:17:54.766106 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762175 5953 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:17:54.766106 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762179 5953 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:17:54.766106 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762183 5953 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:17:54.766106 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762187 5953 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:17:54.766106 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762721 5953 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:17:54.766106 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762728 5953 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:17:54.766106 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762731 5953 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:17:54.766106 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762734 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:17:54.766106 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762737 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:17:54.766106 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762740 5953 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:17:54.766106 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762743 5953 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:17:54.766559 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762746 5953 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:17:54.766559 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762749 5953 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:17:54.766559 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762752 5953 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:17:54.766559 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762754 5953 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:17:54.766559 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762757 5953 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:17:54.766559 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762760 5953 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:17:54.766559 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762762 5953 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:17:54.766559 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762765 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:17:54.766559 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762767 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:17:54.766559 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762770 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:17:54.766559 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762772 5953 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:17:54.766559 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762775 5953 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:17:54.766559 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762778 5953 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:17:54.766559 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762780 5953 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:17:54.766559 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762783 5953 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:17:54.766559 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762788 5953 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:17:54.766559 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762791 5953 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:17:54.766559 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762795 5953 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:17:54.766559 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762798 5953 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:17:54.767046 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762800 5953 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:17:54.767046 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762803 5953 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:17:54.767046 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762806 5953 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:17:54.767046 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762808 5953 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:17:54.767046 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762811 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:17:54.767046 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762814 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:17:54.767046 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762818 5953 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:17:54.767046 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762822 5953 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:17:54.767046 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762825 5953 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:17:54.767046 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762828 5953 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:17:54.767046 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762831 5953 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:17:54.767046 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762834 5953 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:17:54.767046 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762837 5953 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:17:54.767046 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762840 5953 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:17:54.767046 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762843 5953 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:17:54.767046 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762846 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:17:54.767046 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762848 5953 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:17:54.767046 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762851 5953 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:17:54.767046 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762853 5953 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:17:54.767502 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762856 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:17:54.767502 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762858 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:17:54.767502 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762861 5953 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:17:54.767502 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762863 5953 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:17:54.767502 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762866 5953 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:17:54.767502 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762869 5953 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:17:54.767502 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762872 5953 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:17:54.767502 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762874 5953 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:17:54.767502 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762877 5953 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:17:54.767502 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762880 5953 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:17:54.767502 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762883 5953 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:17:54.767502 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762886 5953 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:17:54.767502 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762895 5953 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:17:54.767502 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762898 5953 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:17:54.767502 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762900 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:17:54.767502 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762903 5953 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:17:54.767502 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762905 5953 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:17:54.767502 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762908 5953 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:17:54.767502 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762916 5953 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:17:54.767502 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762919 5953 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:17:54.768002 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762921 5953 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:17:54.768002 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762924 5953 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:17:54.768002 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762926 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:17:54.768002 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762929 5953 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:17:54.768002 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762932 5953 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:17:54.768002 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762934 5953 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:17:54.768002 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762937 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:17:54.768002 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762940 5953 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:17:54.768002 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762942 5953 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:17:54.768002 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762945 5953 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:17:54.768002 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762947 5953 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:17:54.768002 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762950 5953 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:17:54.768002 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762954 5953 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:17:54.768002 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762958 5953 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:17:54.768002 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762961 5953 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:17:54.768002 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762964 5953 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:17:54.768002 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762967 5953 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:17:54.768002 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762969 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:17:54.768002 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762972 5953 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:17:54.768002 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762974 5953 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:17:54.768473 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762977 5953 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:17:54.768473 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762981 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:17:54.768473 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762983 5953 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:17:54.768473 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762986 5953 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:17:54.768473 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762989 5953 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:17:54.768473 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.762996 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:17:54.768473 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763000 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:17:54.768473 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763003 5953 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:17:54.768473 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763005 5953 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:17:54.768473 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763008 5953 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:17:54.768473 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763011 5953 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:17:54.768473 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763014 5953 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:17:54.768473 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763017 5953 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:17:54.768473 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763019 5953 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:17:54.768473 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763022 5953 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:17:54.768473 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763025 5953 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:17:54.768473 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763028 5953 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:17:54.768473 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763031 5953 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:17:54.768473 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763033 5953 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:17:54.768983 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763036 5953 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:17:54.768983 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763038 5953 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:17:54.768983 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763135 5953 flags.go:64] FLAG: --address="0.0.0.0" Nov 04 03:17:54.768983 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763143 5953 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 04 03:17:54.768983 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763150 5953 flags.go:64] FLAG: --anonymous-auth="true" Nov 04 03:17:54.768983 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763157 5953 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 04 03:17:54.768983 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763161 5953 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 04 03:17:54.768983 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763164 5953 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 04 03:17:54.768983 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763169 5953 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 04 03:17:54.768983 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763173 5953 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 04 03:17:54.768983 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763176 5953 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 04 03:17:54.768983 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763179 5953 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 04 03:17:54.768983 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763183 5953 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 04 03:17:54.768983 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763186 5953 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 04 03:17:54.768983 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763191 5953 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 04 03:17:54.768983 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763194 5953 flags.go:64] FLAG: --cgroup-root="" Nov 04 03:17:54.768983 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763197 5953 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 04 03:17:54.768983 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763200 5953 flags.go:64] FLAG: --client-ca-file="" Nov 04 03:17:54.768983 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763203 5953 flags.go:64] FLAG: --cloud-provider="external" Nov 04 03:17:54.768983 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763212 5953 flags.go:64] FLAG: --cluster-dns="[]" Nov 04 03:17:54.768983 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763217 5953 flags.go:64] FLAG: --cluster-domain="" Nov 04 03:17:54.768983 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763220 5953 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 04 03:17:54.768983 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763223 5953 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 04 03:17:54.768983 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763226 5953 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 04 03:17:54.769549 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763229 5953 flags.go:64] FLAG: --container-log-max-files="5" Nov 04 03:17:54.769549 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763234 5953 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 04 03:17:54.769549 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763237 5953 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 04 03:17:54.769549 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763240 5953 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 04 03:17:54.769549 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763243 5953 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 04 03:17:54.769549 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763246 5953 flags.go:64] FLAG: --contention-profiling="false" Nov 04 03:17:54.769549 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763249 5953 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 04 03:17:54.769549 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763252 5953 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 04 03:17:54.769549 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763256 5953 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 04 03:17:54.769549 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763258 5953 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 04 03:17:54.769549 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763263 5953 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 04 03:17:54.769549 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763266 5953 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 04 03:17:54.769549 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763269 5953 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 04 03:17:54.769549 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763271 5953 flags.go:64] FLAG: --enable-load-reader="false" Nov 04 03:17:54.769549 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763282 5953 flags.go:64] FLAG: --enable-server="true" Nov 04 03:17:54.769549 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763285 5953 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 04 03:17:54.769549 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763298 5953 flags.go:64] FLAG: --event-burst="100" Nov 04 03:17:54.769549 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763302 5953 flags.go:64] FLAG: --event-qps="50" Nov 04 03:17:54.769549 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763305 5953 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 04 03:17:54.769549 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763307 5953 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 04 03:17:54.769549 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763310 5953 flags.go:64] FLAG: --eviction-hard="" Nov 04 03:17:54.769549 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763315 5953 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 04 03:17:54.769549 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763319 5953 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 04 03:17:54.769549 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763323 5953 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 04 03:17:54.769549 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763326 5953 flags.go:64] FLAG: --eviction-soft="" Nov 04 03:17:54.770432 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763329 5953 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 04 03:17:54.770432 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763332 5953 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 04 03:17:54.770432 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763336 5953 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 04 03:17:54.770432 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763345 5953 flags.go:64] FLAG: --experimental-mounter-path="" Nov 04 03:17:54.770432 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763348 5953 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 04 03:17:54.770432 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763351 5953 flags.go:64] FLAG: --fail-swap-on="true" Nov 04 03:17:54.770432 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763354 5953 flags.go:64] FLAG: --feature-gates="" Nov 04 03:17:54.770432 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763359 5953 flags.go:64] FLAG: --file-check-frequency="20s" Nov 04 03:17:54.770432 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763362 5953 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 04 03:17:54.770432 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763365 5953 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 04 03:17:54.770432 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763369 5953 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 04 03:17:54.770432 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763372 5953 flags.go:64] FLAG: --healthz-port="10248" Nov 04 03:17:54.770432 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763375 5953 flags.go:64] FLAG: --help="false" Nov 04 03:17:54.770432 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763378 5953 flags.go:64] FLAG: --hostname-override="ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:17:54.770432 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763381 5953 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 04 03:17:54.770432 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763384 5953 flags.go:64] FLAG: --http-check-frequency="20s" Nov 04 03:17:54.770432 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763387 5953 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 04 03:17:54.770432 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763391 5953 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 04 03:17:54.770432 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763394 5953 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 04 03:17:54.770432 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763397 5953 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 04 03:17:54.770432 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763400 5953 flags.go:64] FLAG: --image-service-endpoint="" Nov 04 03:17:54.770432 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763403 5953 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 04 03:17:54.770432 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763406 5953 flags.go:64] FLAG: --kube-api-burst="100" Nov 04 03:17:54.771336 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763409 5953 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 04 03:17:54.771336 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763413 5953 flags.go:64] FLAG: --kube-api-qps="50" Nov 04 03:17:54.771336 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763415 5953 flags.go:64] FLAG: --kube-reserved="" Nov 04 03:17:54.771336 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763419 5953 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 04 03:17:54.771336 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763422 5953 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 04 03:17:54.771336 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763425 5953 flags.go:64] FLAG: --kubelet-cgroups="" Nov 04 03:17:54.771336 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763429 5953 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 04 03:17:54.771336 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763432 5953 flags.go:64] FLAG: --lock-file="" Nov 04 03:17:54.771336 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763435 5953 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 04 03:17:54.771336 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763438 5953 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 04 03:17:54.771336 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763441 5953 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 04 03:17:54.771336 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763448 5953 flags.go:64] FLAG: --log-json-split-stream="false" Nov 04 03:17:54.771336 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763451 5953 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 04 03:17:54.771336 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763458 5953 flags.go:64] FLAG: --log-text-split-stream="false" Nov 04 03:17:54.771336 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763461 5953 flags.go:64] FLAG: --logging-format="text" Nov 04 03:17:54.771336 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763464 5953 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 04 03:17:54.771336 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763467 5953 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 04 03:17:54.771336 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763470 5953 flags.go:64] FLAG: --manifest-url="" Nov 04 03:17:54.771336 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763473 5953 flags.go:64] FLAG: --manifest-url-header="" Nov 04 03:17:54.771336 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763478 5953 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 04 03:17:54.771336 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763481 5953 flags.go:64] FLAG: --max-open-files="1000000" Nov 04 03:17:54.771336 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763485 5953 flags.go:64] FLAG: --max-pods="110" Nov 04 03:17:54.771336 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763488 5953 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 04 03:17:54.771336 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763491 5953 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 04 03:17:54.771336 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763494 5953 flags.go:64] FLAG: --memory-manager-policy="None" Nov 04 03:17:54.772036 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763497 5953 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 04 03:17:54.772036 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763500 5953 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 04 03:17:54.772036 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763503 5953 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 04 03:17:54.772036 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763506 5953 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 04 03:17:54.772036 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763512 5953 flags.go:64] FLAG: --node-status-max-images="50" Nov 04 03:17:54.772036 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763516 5953 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 04 03:17:54.772036 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763519 5953 flags.go:64] FLAG: --oom-score-adj="-999" Nov 04 03:17:54.772036 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763522 5953 flags.go:64] FLAG: --pod-cidr="" Nov 04 03:17:54.772036 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763525 5953 flags.go:64] FLAG: --pod-manifest-path="" Nov 04 03:17:54.772036 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763528 5953 flags.go:64] FLAG: --pod-max-pids="-1" Nov 04 03:17:54.772036 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763531 5953 flags.go:64] FLAG: --pods-per-core="0" Nov 04 03:17:54.772036 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763534 5953 flags.go:64] FLAG: --port="10250" Nov 04 03:17:54.772036 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763537 5953 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 04 03:17:54.772036 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763542 5953 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0280834f1c12e1f06" Nov 04 03:17:54.772036 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763545 5953 flags.go:64] FLAG: --qos-reserved="" Nov 04 03:17:54.772036 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763548 5953 flags.go:64] FLAG: --read-only-port="10255" Nov 04 03:17:54.772036 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763551 5953 flags.go:64] FLAG: --register-node="true" Nov 04 03:17:54.772036 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763554 5953 flags.go:64] FLAG: --register-with-taints="" Nov 04 03:17:54.772036 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763562 5953 flags.go:64] FLAG: --registry-burst="10" Nov 04 03:17:54.772036 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763565 5953 flags.go:64] FLAG: --registry-qps="5" Nov 04 03:17:54.772036 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763568 5953 flags.go:64] FLAG: --reserved-cpus="" Nov 04 03:17:54.772036 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763572 5953 flags.go:64] FLAG: --reserved-memory="" Nov 04 03:17:54.772036 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763576 5953 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 04 03:17:54.772036 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763580 5953 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 04 03:17:54.772036 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763583 5953 flags.go:64] FLAG: --rotate-certificates="false" Nov 04 03:17:54.772036 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763585 5953 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 04 03:17:54.772788 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763589 5953 flags.go:64] FLAG: --runonce="false" Nov 04 03:17:54.772788 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763592 5953 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 04 03:17:54.772788 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763595 5953 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 04 03:17:54.772788 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763598 5953 flags.go:64] FLAG: --seccomp-default="false" Nov 04 03:17:54.772788 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763601 5953 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 04 03:17:54.772788 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763605 5953 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 04 03:17:54.772788 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763608 5953 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 04 03:17:54.772788 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763611 5953 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 04 03:17:54.772788 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763628 5953 flags.go:64] FLAG: --storage-driver-password="root" Nov 04 03:17:54.772788 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763632 5953 flags.go:64] FLAG: --storage-driver-secure="false" Nov 04 03:17:54.772788 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763635 5953 flags.go:64] FLAG: --storage-driver-table="stats" Nov 04 03:17:54.772788 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763638 5953 flags.go:64] FLAG: --storage-driver-user="root" Nov 04 03:17:54.772788 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763641 5953 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 04 03:17:54.772788 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763645 5953 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 04 03:17:54.772788 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763648 5953 flags.go:64] FLAG: --system-cgroups="" Nov 04 03:17:54.772788 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763651 5953 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 04 03:17:54.772788 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763656 5953 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 04 03:17:54.772788 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763659 5953 flags.go:64] FLAG: --tls-cert-file="" Nov 04 03:17:54.772788 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763662 5953 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 04 03:17:54.772788 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763670 5953 flags.go:64] FLAG: --tls-min-version="" Nov 04 03:17:54.772788 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763672 5953 flags.go:64] FLAG: --tls-private-key-file="" Nov 04 03:17:54.772788 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763675 5953 flags.go:64] FLAG: --topology-manager-policy="none" Nov 04 03:17:54.772788 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763678 5953 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 04 03:17:54.772788 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763681 5953 flags.go:64] FLAG: --topology-manager-scope="container" Nov 04 03:17:54.772788 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763685 5953 flags.go:64] FLAG: --v="2" Nov 04 03:17:54.773389 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763690 5953 flags.go:64] FLAG: --version="false" Nov 04 03:17:54.773389 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763694 5953 flags.go:64] FLAG: --vmodule="" Nov 04 03:17:54.773389 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763698 5953 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 04 03:17:54.773389 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.763702 5953 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 04 03:17:54.773389 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763848 5953 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:17:54.773389 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763852 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:17:54.773389 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763855 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:17:54.773389 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763857 5953 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:17:54.773389 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763860 5953 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:17:54.773389 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763863 5953 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:17:54.773389 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763865 5953 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:17:54.773389 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763868 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:17:54.773389 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763870 5953 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:17:54.773389 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763873 5953 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:17:54.773389 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763875 5953 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:17:54.773389 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763878 5953 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:17:54.773389 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763880 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:17:54.773389 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763883 5953 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:17:54.773389 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763885 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:17:54.773389 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763887 5953 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:17:54.774003 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763890 5953 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:17:54.774003 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763892 5953 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:17:54.774003 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763895 5953 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:17:54.774003 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763897 5953 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:17:54.774003 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763900 5953 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:17:54.774003 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763902 5953 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:17:54.774003 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763905 5953 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:17:54.774003 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763908 5953 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:17:54.774003 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763910 5953 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:17:54.774003 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763913 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:17:54.774003 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763916 5953 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:17:54.774003 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763920 5953 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:17:54.774003 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763922 5953 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:17:54.774003 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763925 5953 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:17:54.774003 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763927 5953 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:17:54.774003 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763930 5953 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:17:54.774003 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763938 5953 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:17:54.774003 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763941 5953 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:17:54.774003 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763943 5953 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:17:54.774003 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763946 5953 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:17:54.774486 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763948 5953 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:17:54.774486 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763951 5953 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:17:54.774486 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763954 5953 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:17:54.774486 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763956 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:17:54.774486 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763959 5953 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:17:54.774486 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763961 5953 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:17:54.774486 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763964 5953 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:17:54.774486 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763966 5953 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:17:54.774486 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763969 5953 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:17:54.774486 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763972 5953 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:17:54.774486 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763974 5953 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:17:54.774486 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763976 5953 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:17:54.774486 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763979 5953 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:17:54.774486 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763981 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:17:54.774486 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763984 5953 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:17:54.774486 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763986 5953 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:17:54.774486 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763989 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:17:54.774486 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763991 5953 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:17:54.774486 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763994 5953 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:17:54.775073 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.763997 5953 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:17:54.775073 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764001 5953 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:17:54.775073 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764004 5953 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:17:54.775073 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764007 5953 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:17:54.775073 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764011 5953 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:17:54.775073 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764014 5953 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:17:54.775073 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764016 5953 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:17:54.775073 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764019 5953 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:17:54.775073 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764021 5953 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:17:54.775073 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764024 5953 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:17:54.775073 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764027 5953 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:17:54.775073 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764030 5953 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:17:54.775073 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764032 5953 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:17:54.775073 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764035 5953 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:17:54.775073 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764038 5953 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:17:54.775073 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764040 5953 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:17:54.775073 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764043 5953 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:17:54.775073 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764045 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:17:54.775073 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764048 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:17:54.775726 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764050 5953 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:17:54.775726 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764053 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:17:54.775726 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764055 5953 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:17:54.775726 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764058 5953 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:17:54.775726 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764061 5953 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:17:54.775726 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764063 5953 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:17:54.775726 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764066 5953 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:17:54.775726 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764068 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:17:54.775726 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764072 5953 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:17:54.775726 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764074 5953 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:17:54.775726 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764077 5953 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:17:54.775726 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764081 5953 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:17:54.775726 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764085 5953 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:17:54.775726 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764088 5953 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:17:54.775726 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764091 5953 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:17:54.775726 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764093 5953 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:17:54.775726 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764097 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:17:54.775726 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764100 5953 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:17:54.775726 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764103 5953 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:17:54.776286 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764106 5953 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:17:54.776286 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764109 5953 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:17:54.776286 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764111 5953 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:17:54.776286 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764114 5953 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:17:54.776286 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764116 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:17:54.776286 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764119 5953 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:17:54.776286 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764122 5953 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:17:54.776286 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764124 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:17:54.776286 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764127 5953 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:17:54.776286 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764130 5953 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:17:54.776286 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764132 5953 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:17:54.776286 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764135 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:17:54.776286 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764137 5953 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:17:54.776286 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.764163 5953 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 03:17:54.776286 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764226 5953 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:17:54.777004 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764231 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:17:54.777004 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764234 5953 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:17:54.777004 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764237 5953 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:17:54.777004 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764240 5953 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:17:54.777004 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764242 5953 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:17:54.777004 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764245 5953 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:17:54.777004 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764248 5953 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:17:54.777004 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764250 5953 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:17:54.777004 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764253 5953 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:17:54.777004 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764255 5953 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:17:54.777004 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764258 5953 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:17:54.777004 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764262 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:17:54.777004 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764264 5953 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:17:54.777004 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764267 5953 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:17:54.777004 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764270 5953 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:17:54.777004 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764275 5953 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:17:54.777004 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764277 5953 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:17:54.777004 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764280 5953 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:17:54.777004 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764282 5953 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:17:54.777004 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764285 5953 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:17:54.777605 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764288 5953 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:17:54.777605 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764290 5953 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:17:54.777605 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764293 5953 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:17:54.777605 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764296 5953 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:17:54.777605 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764299 5953 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:17:54.777605 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764301 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:17:54.777605 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764303 5953 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:17:54.777605 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764306 5953 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:17:54.777605 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764309 5953 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:17:54.777605 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764311 5953 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:17:54.777605 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764314 5953 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:17:54.777605 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764316 5953 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:17:54.777605 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764319 5953 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:17:54.777605 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764321 5953 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:17:54.777605 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764324 5953 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:17:54.777605 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764327 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:17:54.777605 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764329 5953 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:17:54.777605 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764331 5953 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:17:54.777605 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764334 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:17:54.777605 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764336 5953 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:17:54.778180 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764339 5953 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:17:54.778180 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764343 5953 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:17:54.778180 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764347 5953 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:17:54.778180 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764350 5953 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:17:54.778180 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764353 5953 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:17:54.778180 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764355 5953 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:17:54.778180 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764358 5953 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:17:54.778180 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764364 5953 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:17:54.778180 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764366 5953 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:17:54.778180 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764369 5953 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:17:54.778180 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764371 5953 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:17:54.778180 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764374 5953 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:17:54.778180 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764377 5953 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:17:54.778180 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764379 5953 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:17:54.778180 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764382 5953 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:17:54.778180 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764384 5953 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:17:54.778180 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764388 5953 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:17:54.778180 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764390 5953 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:17:54.778180 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764393 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:17:54.778764 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764395 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:17:54.778764 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764398 5953 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:17:54.778764 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764401 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:17:54.778764 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764403 5953 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:17:54.778764 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764407 5953 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:17:54.778764 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764410 5953 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:17:54.778764 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764412 5953 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:17:54.778764 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764415 5953 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:17:54.778764 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764417 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:17:54.778764 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764420 5953 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:17:54.778764 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764423 5953 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:17:54.778764 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764425 5953 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:17:54.778764 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764429 5953 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:17:54.778764 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764432 5953 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:17:54.778764 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764435 5953 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:17:54.778764 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764438 5953 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:17:54.778764 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764440 5953 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:17:54.778764 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764443 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:17:54.778764 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764445 5953 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:17:54.779352 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764448 5953 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:17:54.779352 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764453 5953 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:17:54.779352 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764455 5953 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:17:54.779352 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764458 5953 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:17:54.779352 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764461 5953 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:17:54.779352 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764463 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:17:54.779352 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764466 5953 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:17:54.779352 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764468 5953 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:17:54.779352 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764471 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:17:54.779352 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764474 5953 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:17:54.779352 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764476 5953 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:17:54.779352 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764479 5953 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:17:54.779352 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764482 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:17:54.779352 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764485 5953 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:17:54.779352 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764488 5953 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:17:54.779352 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764490 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:17:54.779352 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764493 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:17:54.779352 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764495 5953 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:17:54.779352 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764498 5953 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:17:54.779901 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764500 5953 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:17:54.779901 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764503 5953 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:17:54.779901 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764506 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:17:54.779901 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764508 5953 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:17:54.779901 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764511 5953 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:17:54.779901 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764513 5953 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:17:54.779901 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764516 5953 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:17:54.779901 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.764518 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:17:54.779901 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.769161 5953 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 04 03:17:54.779901 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.769178 5953 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 04 03:17:54.779901 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.769195 5953 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 04 03:17:54.779901 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.769200 5953 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 04 03:17:54.779901 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769272 5953 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:17:54.779901 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769277 5953 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:17:54.779901 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769281 5953 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:17:54.779901 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769284 5953 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:17:54.779901 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769288 5953 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:17:54.779901 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769290 5953 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:17:54.779901 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769293 5953 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:17:54.780687 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769296 5953 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:17:54.780687 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769299 5953 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:17:54.780687 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769302 5953 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:17:54.780687 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769304 5953 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:17:54.780687 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769307 5953 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:17:54.780687 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769310 5953 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:17:54.780687 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769312 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:17:54.780687 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769315 5953 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:17:54.780687 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769317 5953 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:17:54.780687 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769319 5953 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:17:54.780687 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769322 5953 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:17:54.780687 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769325 5953 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:17:54.780687 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769327 5953 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:17:54.780687 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769330 5953 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:17:54.780687 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769332 5953 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:17:54.780687 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769335 5953 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:17:54.780687 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769337 5953 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:17:54.780687 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769340 5953 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:17:54.780687 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769343 5953 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:17:54.781448 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769345 5953 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:17:54.781448 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769348 5953 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:17:54.781448 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769351 5953 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:17:54.781448 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769359 5953 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:17:54.781448 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769362 5953 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:17:54.781448 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769364 5953 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:17:54.781448 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769367 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:17:54.781448 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769369 5953 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:17:54.781448 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769372 5953 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:17:54.781448 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769375 5953 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:17:54.781448 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769378 5953 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:17:54.781448 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769380 5953 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:17:54.781448 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769383 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:17:54.781448 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769385 5953 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:17:54.781448 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769389 5953 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:17:54.781448 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769393 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:17:54.781448 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769397 5953 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:17:54.781448 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769400 5953 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:17:54.781448 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769402 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:17:54.782288 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769404 5953 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:17:54.782288 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769407 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:17:54.782288 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769410 5953 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:17:54.782288 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769412 5953 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:17:54.782288 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769414 5953 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:17:54.782288 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769417 5953 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:17:54.782288 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769419 5953 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:17:54.782288 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769422 5953 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:17:54.782288 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769424 5953 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:17:54.782288 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769427 5953 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:17:54.782288 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769429 5953 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:17:54.782288 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769432 5953 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:17:54.782288 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769434 5953 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:17:54.782288 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769437 5953 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:17:54.782288 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769439 5953 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:17:54.782288 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769442 5953 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:17:54.782288 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769444 5953 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:17:54.782288 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769452 5953 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:17:54.782288 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769455 5953 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:17:54.783150 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769458 5953 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:17:54.783150 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769460 5953 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:17:54.783150 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769464 5953 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:17:54.783150 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769469 5953 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:17:54.783150 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769472 5953 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:17:54.783150 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769474 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:17:54.783150 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769477 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:17:54.783150 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769479 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:17:54.783150 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769482 5953 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:17:54.783150 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769484 5953 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:17:54.783150 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769487 5953 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:17:54.783150 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769489 5953 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:17:54.783150 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769492 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:17:54.783150 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769494 5953 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:17:54.783150 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769497 5953 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:17:54.783150 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769500 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:17:54.783150 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769502 5953 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:17:54.783150 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769505 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:17:54.783150 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769507 5953 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:17:54.783150 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769510 5953 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:17:54.784023 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769512 5953 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:17:54.784023 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769515 5953 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:17:54.784023 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769517 5953 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:17:54.784023 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769520 5953 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:17:54.784023 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769522 5953 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:17:54.784023 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769525 5953 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:17:54.784023 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769527 5953 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:17:54.784023 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769530 5953 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:17:54.784023 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769532 5953 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:17:54.784023 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769535 5953 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:17:54.784023 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769537 5953 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:17:54.784023 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769545 5953 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:17:54.784023 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769548 5953 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:17:54.784023 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769550 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:17:54.784023 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769553 5953 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:17:54.784023 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769556 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:17:54.784023 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769559 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:17:54.784023 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769561 5953 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:17:54.784023 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769564 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:17:54.784023 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769566 5953 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:17:54.784866 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769569 5953 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:17:54.784866 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.769571 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:17:54.784866 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.769600 5953 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 03:17:54.784866 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770427 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:17:54.784866 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770443 5953 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:17:54.784866 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770449 5953 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:17:54.784866 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770454 5953 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:17:54.784866 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770458 5953 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:17:54.784866 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770463 5953 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:17:54.784866 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770467 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:17:54.784866 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770471 5953 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:17:54.784866 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770475 5953 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:17:54.784866 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770480 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:17:54.784866 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770491 5953 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:17:54.784866 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770495 5953 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:17:54.784866 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770500 5953 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:17:54.785544 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770504 5953 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:17:54.785544 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770508 5953 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:17:54.785544 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770512 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:17:54.785544 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770516 5953 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:17:54.785544 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770520 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:17:54.785544 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770525 5953 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:17:54.785544 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770529 5953 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:17:54.785544 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770533 5953 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:17:54.785544 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770595 5953 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:17:54.785544 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770601 5953 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:17:54.785544 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770612 5953 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:17:54.785544 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770634 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:17:54.785544 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770639 5953 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:17:54.785544 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770643 5953 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:17:54.785544 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770648 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:17:54.785544 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770652 5953 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:17:54.785544 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770657 5953 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:17:54.785544 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770662 5953 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:17:54.785544 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770667 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:17:54.785544 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770672 5953 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:17:54.786401 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770677 5953 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:17:54.786401 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770682 5953 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:17:54.786401 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770686 5953 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:17:54.786401 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770695 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:17:54.786401 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770699 5953 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:17:54.786401 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770704 5953 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:17:54.786401 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770709 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:17:54.786401 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770714 5953 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:17:54.786401 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770719 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:17:54.786401 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770724 5953 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:17:54.786401 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770729 5953 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:17:54.786401 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770734 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:17:54.786401 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770739 5953 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:17:54.786401 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770745 5953 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:17:54.786401 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770750 5953 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:17:54.786401 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770760 5953 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:17:54.786401 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770765 5953 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:17:54.786401 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770770 5953 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:17:54.786401 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770775 5953 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:17:54.786401 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770779 5953 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:17:54.787547 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770784 5953 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:17:54.787547 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770819 5953 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:17:54.787547 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770827 5953 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:17:54.787547 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770834 5953 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:17:54.787547 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770840 5953 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:17:54.787547 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770844 5953 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:17:54.787547 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770849 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:17:54.787547 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770858 5953 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:17:54.787547 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770862 5953 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:17:54.787547 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770866 5953 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:17:54.787547 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770871 5953 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:17:54.787547 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770876 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:17:54.787547 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770880 5953 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:17:54.787547 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770885 5953 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:17:54.787547 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770889 5953 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:17:54.787547 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770893 5953 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:17:54.787547 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770897 5953 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:17:54.787547 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770902 5953 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:17:54.787547 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770906 5953 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:17:54.788374 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770910 5953 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:17:54.788374 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770920 5953 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:17:54.788374 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770923 5953 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:17:54.788374 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770928 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:17:54.788374 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770932 5953 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:17:54.788374 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770936 5953 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:17:54.788374 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770941 5953 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:17:54.788374 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770945 5953 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:17:54.788374 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770950 5953 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:17:54.788374 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770955 5953 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:17:54.788374 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770959 5953 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:17:54.788374 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770964 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:17:54.788374 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770968 5953 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:17:54.788374 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770973 5953 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:17:54.788374 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.770983 5953 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:17:54.788374 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771055 5953 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:17:54.788374 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771061 5953 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:17:54.788374 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771065 5953 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:17:54.788374 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771070 5953 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:17:54.789179 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771075 5953 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:17:54.789179 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771079 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:17:54.789179 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771084 5953 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:17:54.789179 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771088 5953 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:17:54.789179 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771092 5953 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:17:54.789179 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771097 5953 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:17:54.789179 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771101 5953 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:17:54.789179 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771106 5953 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:17:54.789179 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771115 5953 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:17:54.789179 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771119 5953 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:17:54.789179 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771126 5953 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:17:54.789179 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771133 5953 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:17:54.789179 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771138 5953 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:17:54.789179 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771143 5953 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:17:54.789179 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771148 5953 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:17:54.789179 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.771254 5953 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 03:17:54.789926 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771438 5953 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:17:54.789926 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771449 5953 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:17:54.789926 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771476 5953 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:17:54.789926 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771480 5953 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:17:54.789926 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771484 5953 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:17:54.789926 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771600 5953 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:17:54.789926 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771604 5953 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:17:54.789926 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771607 5953 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:17:54.789926 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771612 5953 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:17:54.789926 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771737 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:17:54.789926 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771741 5953 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:17:54.789926 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771744 5953 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:17:54.789926 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771747 5953 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:17:54.789926 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771750 5953 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:17:54.789926 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771753 5953 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:17:54.789926 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771755 5953 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:17:54.789926 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771758 5953 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:17:54.789926 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771761 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:17:54.789926 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771764 5953 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:17:54.789926 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771768 5953 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:17:54.790830 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771771 5953 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:17:54.790830 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771774 5953 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:17:54.790830 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771776 5953 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:17:54.790830 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771779 5953 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:17:54.790830 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771782 5953 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:17:54.790830 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771785 5953 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:17:54.790830 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771787 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:17:54.790830 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771790 5953 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:17:54.790830 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771792 5953 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:17:54.790830 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771795 5953 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:17:54.790830 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771798 5953 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:17:54.790830 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771800 5953 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:17:54.790830 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771803 5953 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:17:54.790830 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771806 5953 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:17:54.790830 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771808 5953 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:17:54.790830 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771813 5953 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:17:54.790830 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771817 5953 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:17:54.790830 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771821 5953 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:17:54.790830 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771823 5953 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:17:54.791811 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771826 5953 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:17:54.791811 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771829 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:17:54.791811 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771832 5953 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:17:54.791811 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771835 5953 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:17:54.791811 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771837 5953 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:17:54.791811 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771840 5953 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:17:54.791811 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771843 5953 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:17:54.791811 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771846 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:17:54.791811 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771849 5953 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:17:54.791811 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771852 5953 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:17:54.791811 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771855 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:17:54.791811 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771858 5953 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:17:54.791811 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771861 5953 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:17:54.791811 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771864 5953 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:17:54.791811 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771866 5953 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:17:54.791811 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771869 5953 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:17:54.791811 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771871 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:17:54.791811 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771874 5953 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:17:54.791811 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771877 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:17:54.791811 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771879 5953 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:17:54.792767 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771882 5953 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:17:54.792767 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771885 5953 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:17:54.792767 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771888 5953 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:17:54.792767 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771890 5953 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:17:54.792767 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771893 5953 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:17:54.792767 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771896 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:17:54.792767 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771899 5953 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:17:54.792767 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771901 5953 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:17:54.792767 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771905 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:17:54.792767 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771909 5953 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:17:54.792767 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771913 5953 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:17:54.792767 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771917 5953 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:17:54.792767 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771921 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:17:54.792767 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771925 5953 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:17:54.792767 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771929 5953 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:17:54.792767 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771931 5953 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:17:54.792767 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771936 5953 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:17:54.792767 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771939 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:17:54.792767 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771941 5953 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:17:54.792767 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771944 5953 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:17:54.793701 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771947 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:17:54.793701 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771949 5953 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:17:54.793701 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771953 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:17:54.793701 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771955 5953 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:17:54.793701 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771958 5953 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:17:54.793701 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771963 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:17:54.793701 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771966 5953 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:17:54.793701 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771969 5953 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:17:54.793701 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771972 5953 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:17:54.793701 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771974 5953 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:17:54.793701 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771977 5953 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:17:54.793701 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771979 5953 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:17:54.793701 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771982 5953 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:17:54.793701 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771985 5953 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:17:54.793701 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771987 5953 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:17:54.793701 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771990 5953 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:17:54.793701 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771994 5953 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:17:54.793701 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.771999 5953 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:17:54.793701 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.772002 5953 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:17:54.794690 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.772005 5953 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:17:54.794690 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.772008 5953 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:17:54.794690 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.772010 5953 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:17:54.794690 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.772013 5953 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:17:54.794690 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.772016 5953 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:17:54.794690 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.772019 5953 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:17:54.794690 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.772023 5953 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:17:54.794690 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:17:54.772025 5953 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:17:54.794690 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.772096 5953 server.go:957] "Client rotation is on, will bootstrap in background" Nov 04 03:17:54.794690 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.773095 5953 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 04 03:17:54.794690 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.773155 5953 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 04 03:17:54.794690 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.773327 5953 server.go:1014] "Starting client certificate rotation" Nov 04 03:17:54.794690 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.773442 5953 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 03:17:54.794690 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.773504 5953 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-03 03:11:42 +0000 UTC" deadline="2028-06-18 14:51:14.031255416 +0000 UTC" Nov 04 03:17:54.794690 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.773545 5953 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14219h33m19.257713725s" Nov 04 03:17:54.794690 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.773833 5953 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 03:17:54.795434 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.774939 5953 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 03:17:54.795434 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.776325 5953 log.go:25] "Validated CRI v1 runtime API" Nov 04 03:17:54.795434 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.778482 5953 log.go:25] "Validated CRI v1 image API" Nov 04 03:17:54.795434 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.779428 5953 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 04 03:17:54.795434 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.783041 5953 fs.go:135] Filesystem UUIDs: map[2acf8da6-c497-48ca-9887-7c0f8c314692:/dev/nvme0n1p3 732e0981-1f45-46c0-8881-cb4c01bb118c:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 04 03:17:54.795734 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.783070 5953 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1846dbcff1a501528f398dfde9d918e157bceba20e37ce363fa71726ac064d1e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1846dbcff1a501528f398dfde9d918e157bceba20e37ce363fa71726ac064d1e/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1a6cc4f32dbf89779adc13674239ca74ae1dd88dd77759d62708826570f184b4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1a6cc4f32dbf89779adc13674239ca74ae1dd88dd77759d62708826570f184b4/userdata/shm major:0 minor:204 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1da1db8c80e0d30a8d1ff45a275e72414ff80b95ee8e6dba298e60a9807c7740/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1da1db8c80e0d30a8d1ff45a275e72414ff80b95ee8e6dba298e60a9807c7740/userdata/shm major:0 minor:472 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1f357050039df0c66f46c3927903c9f913ef0340757c5043ead5991a62a0963b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1f357050039df0c66f46c3927903c9f913ef0340757c5043ead5991a62a0963b/userdata/shm major:0 minor:344 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2b9d93ac2db03e8dfc74e8608d2ceaeec92c843309e60201ee4057bc5fa939bb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2b9d93ac2db03e8dfc74e8608d2ceaeec92c843309e60201ee4057bc5fa939bb/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5661b48df391450ec18633edf94716bde74a07933d0ef030521ad754a4909bf6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5661b48df391450ec18633edf94716bde74a07933d0ef030521ad754a4909bf6/userdata/shm major:0 minor:252 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6132bb22558201eafc6fe42434f09c31763011a7226766084223d4783a699f8f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6132bb22558201eafc6fe42434f09c31763011a7226766084223d4783a699f8f/userdata/shm major:0 minor:304 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/63559e597407dae564a620e85c94cbf6318bd09f5c75313d0e6fdd9fba2f126a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/63559e597407dae564a620e85c94cbf6318bd09f5c75313d0e6fdd9fba2f126a/userdata/shm major:0 minor:470 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/64803c7435bf8b87b711ba3436cb3778c5f7bebc554c6d345f97132b5a0c8104/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/64803c7435bf8b87b711ba3436cb3778c5f7bebc554c6d345f97132b5a0c8104/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6ec7ae50a4a7d2295b6890493085aaf6410dcfdc92d906ba9d10043571c9812b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6ec7ae50a4a7d2295b6890493085aaf6410dcfdc92d906ba9d10043571c9812b/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/75a2b016aaf3f6408b28d110698ecd053b2ae2ecbaa8aa7bcb68e1e6c2c30068/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/75a2b016aaf3f6408b28d110698ecd053b2ae2ecbaa8aa7bcb68e1e6c2c30068/userdata/shm major:0 minor:473 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/760c6e388680b0e6510ec7d509e014d206711baf48b8309f94bb523169a34ede/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/760c6e388680b0e6510ec7d509e014d206711baf48b8309f94bb523169a34ede/userdata/shm major:0 minor:329 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/791796dc50b8bec54529e842614d63f8768af87659c325aa9ada29014e330c61/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/791796dc50b8bec54529e842614d63f8768af87659c325aa9ada29014e330c61/userdata/shm major:0 minor:463 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/79ce3567247d25712b2d4037523f0437b4da86f3d90d0bac8859b00d5cf3eeee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/79ce3567247d25712b2d4037523f0437b4da86f3d90d0bac8859b00d5cf3eeee/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/93b3cbc9035ec7cec6ac37617c10933127bb5177af65587c6898c97cb381c6cd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/93b3cbc9035ec7cec6ac37617c10933127bb5177af65587c6898c97cb381c6cd/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/988dc3031df7bc31a97c6a25b8e24cda782cef441f820edcda4057223ffb7655/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/988dc3031df7bc31a97c6a25b8e24cda782cef441f820edcda4057223ffb7655/userdata/shm major:0 minor:64 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a54ba8507a55dea18a87dbe7af55c0f565972c855b2bf5b1b97864bc36355c1d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a54ba8507a55dea18a87dbe7af55c0f565972c855b2bf5b1b97864bc36355c1d/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ba74b6466c9db3a2e9b4811b48d639a47993d10088c68fda03f7053e5d099e10/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ba74b6466c9db3a2e9b4811b48d639a47993d10088c68fda03f7053e5d099e10/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bd778358e255e28aa0c96fbbfcca8600ba0832bf0e6a2c63ac455194cecfc0a8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bd778358e255e28aa0c96fbbfcca8600ba0832bf0e6a2c63ac455194cecfc0a8/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c4998297ab19ab972cd9dbc2637cfe3b110924f917064937d89ed0ffc0959fbd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c4998297ab19ab972cd9dbc2637cfe3b110924f917064937d89ed0ffc0959fbd/userdata/shm major:0 minor:357 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c777436c764c665c4bb019ddad6db5c86ce03725b182ddcb23c951590f8595af/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c777436c764c665c4bb019ddad6db5c86ce03725b182ddcb23c951590f8595af/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d334d5ffb746244dc6e3456097b41613e54e96dfd4dbbb4e77995ab0fd416185/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d334d5ffb746244dc6e3456097b41613e54e96dfd4dbbb4e77995ab0fd416185/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d5c3b5befed935528d1a226b03c5e57dd4e4f5bee495cdf947cc098a3657f302/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d5c3b5befed935528d1a226b03c5e57dd4e4f5bee495cdf947cc098a3657f302/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ea615393cda142a2d94cdcc77c191ac97a95d132f96c4c9a6636513c52ea39f3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ea615393cda142a2d94cdcc77c191ac97a95d132f96c4c9a6636513c52ea39f3/userdata/shm major:0 minor:346 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f99e820db2e951c4e4a712acb3cd6c6d622e86bc03be5ed241a895818fb6e00c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f99e820db2e951c4e4a712acb3cd6c6d622e86bc03be5ed241a895818fb6e00c/userdata/shm major:0 minor:245 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16659cac-4c32-4a94-90aa-8be0c6be33b2/volumes/kubernetes.io~projected/kube-api-access-fcpzg:{mountpoint:/var/lib/kubelet/pods/16659cac-4c32-4a94-90aa-8be0c6be33b2/volumes/kubernetes.io~projected/kube-api-access-fcpzg major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16659cac-4c32-4a94-90aa-8be0c6be33b2/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/16659cac-4c32-4a94-90aa-8be0c6be33b2/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16659cac-4c32-4a94-90aa-8be0c6be33b2/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/16659cac-4c32-4a94-90aa-8be0c6be33b2/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1aa3f04c-b684-4426-8d46-a36d8c4e77be/volumes/kubernetes.io~projected/kube-api-access-zdrr8:{mountpoint:/var/lib/kubelet/pods/1aa3f04c-b684-4426-8d46-a36d8c4e77be/volumes/kubernetes.io~projected/kube-api-access-zdrr8 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1aa3f04c-b684-4426-8d46-a36d8c4e77be/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/1aa3f04c-b684-4426-8d46-a36d8c4e77be/volumes/kubernetes.io~secret/metrics-certs major:0 minor:467 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f6b477e-a0fd-49c8-896d-c268a6c06d3f/volumes/kubernetes.io~projected/kube-api-access-vjwjp:{mountpoint:/var/lib/kubelet/pods/1f6b477e-a0fd-49c8-896d-c268a6c06d3f/volumes/kubernetes.io~projected/kube-api-access-vjwjp major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/326a1269-c427-4470-86a6-70ee348c8791/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/326a1269-c427-4470-86a6-70ee348c8791/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/326a1269-c427-4470-86a6-70ee348c8791/volumes/kubernetes.io~projected/kube-api-access-8qzd8:{mountpoint:/var/lib/kubelet/pods/326a1269-c427-4470-86a6-70ee348c8791/volumes/kubernetes.io~projected/kube-api-access-8qzd8 major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/326a1269-c427-4470-86a6-70ee348c8791/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/326a1269-c427-4470-86a6-70ee348c8791/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44f960f8-1c9c-4a1b-a422-5477a6d49316/volumes/kubernetes.io~projected/kube-api-access-szld4:{mountpoint:/var/lib/kubelet/pods/44f960f8-1c9c-4a1b-a422-5477a6d49316/volumes/kubernetes.io~projected/kube-api-access-szld4 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44f960f8-1c9c-4a1b-a422-5477a6d49316/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/44f960f8-1c9c-4a1b-a422-5477a6d49316/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ab83526-361f-41de-b480-79fc45d7373a/volumes/kubernetes.io~projected/kube-api-access-94vqb:{mountpoint:/var/lib/kubelet/pods/4ab83526-361f-41de-b480-79fc45d7373a/volumes/kubernetes.io~projected/kube-api-access-94vqb major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ab83526-361f-41de-b480-79fc45d7373a/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/4ab83526-361f-41de-b480-79fc45d7373a/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ab83526-361f-41de-b480-79fc45d7373a/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/4ab83526-361f-41de-b480-79fc45d7373a/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ab83526-361f-41de-b480-79fc45d7373a/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/4ab83526-361f-41de-b480-79fc45d7373a/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ab83526-361f-41de-b480-79fc45d7373a/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/4ab83526-361f-41de-b480-79fc45d7373a/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/57272665-8b2d-4854-bcb6-98f662716f4c/volumes/kubernetes.io~projected/kube-api-access-s7c4r:{mountpoint:/var/lib/kubelet/pods/57272665-8b2d-4854-bcb6-98f662716f4c/volumes/kubernetes.io~projected/kube-api-access-s7c4r major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/57272665-8b2d-4854-bcb6-98f662716f4c/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/57272665-8b2d-4854-bcb6-98f662716f4c/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/57272665-8b2d-4854-bcb6-98f662716f4c/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/57272665-8b2d-4854-bcb6-98f662716f4c/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/719911b7-0c11-4c79-bb86-fdc043d9f9ab/volumes/kubernetes.io~projected/kube-api-access-n7xg7:{mountpoint:/var/lib/kubelet/pods/719911b7-0c11-4c79-bb86-fdc043d9f9ab/volumes/kubernetes.io~projected/kube-api-access-n7xg7 major:0 minor:469 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/834b5587-1820-4089-a925-ab992fe654b4/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/834b5587-1820-4089-a925-ab992fe654b4/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:468 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/923f8598-9df4-46bb-8da0-73442c97522f/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/923f8598-9df4-46bb-8da0-73442c97522f/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab/volumes/kubernetes.io~projected/kube-api-access-ngsgw:{mountpoint:/var/lib/kubelet/pods/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab/volumes/kubernetes.io~projected/kube-api-access-ngsgw major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9acbca43-6baa-4f8f-9eee-eb08dc731588/volumes/kubernetes.io~projected/kube-api-access-s869g:{mountpoint:/var/lib/kubelet/pods/9acbca43-6baa-4f8f-9eee-eb08dc731588/volumes/kubernetes.io~projected/kube-api-access-s869g major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9acbca43-6baa-4f8f-9eee-eb08dc731588/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/9acbca43-6baa-4f8f-9eee-eb08dc731588/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f7d1536-9b78-4f13-ae4e-349db78d213e/volumes/kubernetes.io~projected/kube-api-access-kcw55:{mountpoint:/var/lib/kubelet/pods/9f7d1536-9b78-4f13-ae4e-349db78d213e/volumes/kubernetes.io~projected/kube-api-access-kcw55 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a040e5d8-7316-4a37-a045-9eb7c2eaf607/volumes/kubernetes.io~projected/kube-api-access-pvm8t:{mountpoint:/var/lib/kubelet/pods/a040e5d8-7316-4a37-a045-9eb7c2eaf607/volumes/kubernetes.io~projected/kube-api-access-pvm8t major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a040e5d8-7316-4a37-a045-9eb7c2eaf607/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/a040e5d8-7316-4a37-a045-9eb7c2eaf607/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a040e5d8-7316-4a37-a045-9eb7c2eaf607/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/a040e5d8-7316-4a37-a045-9eb7c2eaf607/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271/volumes/kubernetes.io~projected/kube-api-access-tbcgk:{mountpoint:/var/lib/kubelet/pods/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271/volumes/kubernetes.io~projected/kube-api-access-tbcgk major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c63fa297-f5dd-4c87-aa7b-f691ad61cb45/volumes/kubernetes.io~projected/kube-api-access-j2ftw:{mountpoint:/var/lib/kubelet/pods/c63fa297-f5dd-4c87-aa7b-f691ad61cb45/volumes/kubernetes.io~projected/kube-api-access-j2ftw major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c63fa297-f5dd-4c87-aa7b-f691ad61cb45/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/c63fa297-f5dd-4c87-aa7b-f691ad61cb45/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca254b76-1d53-44aa-81e5-e40a81265145/volumes/kubernetes.io~projected/kube-api-access-fqc8z:{mountpoint:/var/lib/kubelet/pods/ca254b76-1d53-44aa-81e5-e40a81265145/volumes/kubernetes.io~projected/kube-api-access-fqc8z major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca5887c7-01d6-44ee-ac30-8b4374663d3e/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/ca5887c7-01d6-44ee-ac30-8b4374663d3e/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca5887c7-01d6-44ee-ac30-8b4374663d3e/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/ca5887c7-01d6-44ee-ac30-8b4374663d3e/volumes/kubernetes.io~empty-dir/tmp major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca5887c7-01d6-44ee-ac30-8b4374663d3e/volumes/kubernetes.io~projected/kube-api-access-q24vc:{mountpoint:/var/lib/kubelet/pods/ca5887c7-01d6-44ee-ac30-8b4374663d3e/volumes/kubernetes.io~projected/kube-api-access-q24vc major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd03c8ce-341a-4afe-93a1-2280392dd4f8/volumes/kubernetes.io~projected/kube-api-access-gtnzk:{mountpoint:/var/lib/kubelet/pods/cd03c8ce-341a-4afe-93a1-2280392dd4f8/volumes/kubernetes.io~projected/kube-api-access-gtnzk major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd03c8ce-341a-4afe-93a1-2280392dd4f8/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/cd03c8ce-341a-4afe-93a1-2280392dd4f8/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volume-subpaths/web-config/alertmanager/9 major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volume-subpaths/web-config/config-reloader/6 major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volumes/kubernetes.io~empty-dir/config-out major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volumes/kubernetes.io~projected/kube-api-access-f99pm:{mountpoint:/var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volumes/kubernetes.io~projected/kube-api-access-f99pm major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volumes/kubernetes.io~projected/tls-assets major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volumes/kubernetes.io~secret/config-volume major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volumes/kubernetes.io~secret/web-config major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6ebb345-a666-4ece-9afe-c1be0a44867b/volumes/kubernetes.io~projected/kube-api-access-5mr7j:{mountpoint:/var/lib/kubelet/pods/d6ebb345-a666-4ece-9afe-c1be0a44867b/volumes/kubernetes.io~projected/kube-api-access-5mr7j major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6ebb345-a666-4ece-9afe-c1be0a44867b/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/d6ebb345-a666-4ece-9afe-c1be0a44867b/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6ebb345-a666-4ece-9afe-c1be0a44867b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/d6ebb345-a666-4ece-9afe-c1be0a44867b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6ebb345-a666-4ece-9afe-c1be0a44867b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/d6ebb345-a666-4ece-9afe-c1be0a44867b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6ebb345-a666-4ece-9afe-c1be0a44867b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/d6ebb345-a666-4ece-9afe-c1be0a44867b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6ebb345-a666-4ece-9afe-c1be0a44867b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/d6ebb345-a666-4ece-9afe-c1be0a44867b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6ebb345-a666-4ece-9afe-c1be0a44867b/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/d6ebb345-a666-4ece-9afe-c1be0a44867b/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volume-subpaths/web-config/config-reloader/2 major:0 minor:452 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:452 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volume-subpaths/web-config/prometheus/15 major:0 minor:452 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volumes/kubernetes.io~empty-dir/config-out major:0 minor:457 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volumes/kubernetes.io~projected/kube-api-access-9rmmf:{mountpoint:/var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volumes/kubernetes.io~projected/kube-api-access-9rmmf major:0 minor:462 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volumes/kubernetes.io~projected/tls-assets major:0 minor:448 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volumes/kubernetes.io~secret/config major:0 minor:454 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:459 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:456 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:453 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:461 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:458 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:460 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:455 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volumes/kubernetes.io~secret/web-config major:0 minor:452 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5e3933f-6610-46a7-934a-8ce205da98d9/volumes/kubernetes.io~projected/kube-api-access-zmxc6:{mountpoint:/var/lib/kubelet/pods/e5e3933f-6610-46a7-934a-8ce205da98d9/volumes/kubernetes.io~projected/kube-api-access-zmxc6 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb15c957-99e8-4c15-b382-788025661809/volumes/kubernetes.io~projected/kube-api-access-cdtwq:{mountpoint:/var/lib/kubelet/pods/fb15c957-99e8-4c15-b382-788025661809/volumes/kubernetes.io~projected/kube-api-access-cdtwq major:0 minor:55 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/3e3dd325ce35ba5b7457653dc0480761d448b4fd03887eb70eea0477e7a03261/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/560c5592559fdfa5d4d135e92dbc4c948d6e129874df2ac1248970dfdaee9299/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/2760955ecb2a9bff250451118cf0d5edda3a71d439477dd05f06c8771e34fa6b/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/6b99391200278ab3f24155cc499909a96fda93885210d86de10625e13a28888d/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/b3d8f7c2953826cf9b6ddea0d8ad373a54ca1fcdd570132d3b843a4f9c4106e6/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/ca420b381ffc7999c0241a73f17b3716c579ec3ec7ffa2c42600c5f76df85401/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/bcab6d0e15df1e59f11d3352910ba09436b51e0e0d66c090b90220eb5002af35/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/a03cb49cb307f8547cc87e35d6ae567240dd6833e01ff8f71193711a205b2d9a/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-130:{mountpoint:/var/lib/containers/storage/overlay/5d761b0e25797b841975f8cd87817a64cdfb05ff316ed81dd261b695fc09cbb7/merged major:0 minor:130 fsType:overlay blockSize:0} overlay_0-135:{mountpoint:/var/lib/containers/storage/overlay/45be62f4dd89f5a2bf3796eb10511478dfc4f84c3c3974e1f497f14b254a1d45/merged major:0 minor:135 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/3ec3ff1f3391c20a8c19ba47aa7931d44f72e3ce3515ab980c02cd7f2e77c901/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-148:{mountpoint:/var/lib/containers/storage/overlay/ec163d3543e11b2176c40be785439948c1f6eb065c13399fdff145ed200528f6/merged major:0 minor:148 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/3dd87cc0ecd21fef135d3d9e7665d0fc86a733154e1a7e4bcb1f82186a97ceaa/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/919a398277455455148f7e54f84f97090e4dc12508f3d2c09044dd4c97026805/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/1555d6aed2df2674916d6c9f2cbfaa2039f48c8f9d761e7ccef9bd32f23b7647/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/c08e0a079905163301c17eb377e06e1d67a8fafb37c8cdca7532cbe8b93e9672/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/ac84349bcf6ea76a121df0546eb965ec3f85da3365140fb68c7b818f0644a2f1/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/fbc8f13c2d42e15f36ed7190646d159f7d442429f49ebd1f18d28748c7f01dcf/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-179:{mountpoint:/var/lib/containers/storage/overlay/ef3ab519db2a4d12abc049946043f37e0b6b1b7eec9bc3723f39e96316afcede/merged major:0 minor:179 fsType:overlay blockSize:0} overlay_0-184:{mountpoint:/var/lib/containers/storage/overlay/67adb890d659c3297d66c12a63442dc01b5eeb6ee5922c2119b20fd52e738b59/merged major:0 minor:184 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/71752179a44cd3b033f7d795a0d451032f35a5926a70ef50e79eac89589ffdc6/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/cf0898feebcf4cf65dac87281935da9ac4e862c40922f290783b786268898a4a/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/50f61d91c5f67391d76ab0a8ec26b3cb64ee367c84851f7cc5b2bcbedc4f3f2d/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/d3bb5f0c7d19a2ed3224ba9636a99006a6beb760a49099c49e76eb2c054e9d00/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/e2a5ca275a577aed139a9ff49efd767bdb370880ff7fe90b3b029d0d34c1537d/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/7da793e6d777d146ef8a4a72a20890bb0f1585f4ea3f1478e0c201f8c28c23e7/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/8f659e78e2a08949741bbd1ecf8cd6731d64553e160b4a3d7517f7e5c1b71207/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/a702084447aefd0f94168eae9bbe037a5678e43a71989b729d5df9d99bae98e3/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/648d26f49bb5bc2419ad2f028a3584f4fa5f98683d8f8db2afe417bcd730a4ba/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/51d8aa20c9b618068f3c0f6f2520eeba66a79033c8f8d9a865f0888f4121ce8c/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/ce1d61b9afb2c9219471531c4345cc581e854fdc2294dec31ffc598013705516/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/1e946e383a10a64236c54a70ca4c00a9ff294cca19425be629267922875e7719/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/7b2fc9301ef34e4894e8517efb3b57dcaca77498864668c4368a7d4748a40847/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/7bf350ae4a1f812811b94343f2faaca655ceb36ac95b057c95ae3e14e8ebdb12/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/b9f69da64a4e6e8dc05de4447cec3ab0bbd9ed5b0a34138634a2dd6320bdb490/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/b8dadfc99aed77d4e92b5d1f5ee7adeb805cbc6ad8629f240afdaf036ad0725a/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/ed67cb51b99da33801d5cfd90fe604e6da3ec45f42b9ced81e05f1fdf20742dc/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/f31168b99aaa9a94fc4abb364cf38aebced4c53e39a6ff3920f4f9e228be7650/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/e51fb5fcf25dddfcf1f39104a096b1cb94841e05a6a95c3a448e8732096e2341/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/aa14383bc1d480d31c5efae089cf5607f96feea2782bbe539b627c63e22d0fc0/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/6858ef4cc8a23171c0dd07d0b98c55b918ceaf94fe441765b2e697948d3a2dfc/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/59d8573bf1f332c4aab13f2aabfa1dfcf8e6e61ae18869874bf90bbeca2306ae/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/3d01d763866bafc8e5d249016c96dc2f487853a7fad51ff65120b2b38f3eed67/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/f2a73d484c5410fd3aebf42c3147502cc741b8399509fa2190231768011a1924/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/f4366df9e1fa0495d0e03aee44327ae55a516e9887b858428ae274c019cef8cf/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/5ad1a20865579c193d43c8f413a9794da208c0bb7b2dd4afab24508107aae01d/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/774a30401e9b9c0fe81c184840d3e7c7425822c0cebb0a078de9abd4930cb4f2/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/feca25e2fd4a3a01e3b542d219bcb1ad2d2eca194eed2344ea29d549aef81070/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/26e17ecb0f6a0c58566485326dc62af8ab01e96f43ce62720eebd971dd9958ca/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/72c3aeeb66a30511dfd22d09df92eda05d24cf1f8cc6c79f2cdfbed0f2edc6fc/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-379:{mountpoint:/var/lib/containers/storage/overlay/b4efecb4d86be59a83570eff1ffa97fa8fdd4e1c4582d1abe4cb44d92cc2b1ea/merged major:0 minor:379 fsType:overlay blockSize:0} overlay_0-384:{mountpoint:/var/lib/containers/storage/overlay/a8914837d3e2426f73a5c1051d41eebe30c632e2b7394b59079900cc0444dc01/merged major:0 minor:384 fsType:overlay blockSize:0} overlay_0-386:{mountpoint:/var/lib/containers/storage/overlay/dfc852394ba68dfc737b67f4ba05f45b8931beb5964ffd0fdd8c30e80f87f276/merged major:0 minor:386 fsType:overlay blockSize:0} overlay_0-387:{mountpoint:/var/lib/containers/storage/overlay/ec4f3a114f09ac36e8120e45a538d9810a9661e32285f76c0affac667b04f621/merged major:0 minor:387 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/855ad4677ad5a9b89f9c73b811051c5ff65e97a92c42a80407026fffb4497a07/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-403:{mountpoint:/var/lib/containers/storage/overlay/45dd44706b196e1597d4a8e5bfe1b0e2648d0c41753335aa512b3b4f01797965/merged major:0 minor:403 fsType:overlay blockSize:0} overlay_0-404:{mountpoint:/var/lib/containers/storage/overlay/a4cd873e1050603c1d360b3748cfd468bca658dbd209778e4f98ffbfdcd4f912/merged major:0 minor:404 fsType:overlay blockSize:0} overlay_0-416:{mountpoint:/var/lib/containers/storage/overlay/9a0f2fec1f99283f471f9e88bbf9d22774a2a20e547bd351930b40e36ea1335c/merged major:0 minor:416 fsType:overlay blockSize:0} overlay_0-418:{mountpoint:/var/lib/containers/storage/overlay/a7c4ac4e731e6bdecfcbd257e7b16de6c9af3e543d5c2ac1c99f954ece1f979c/merged major:0 minor:418 fsType:overlay blockSize:0} overlay_0-427:{mountpoint:/var/lib/containers/storage/overlay/a3106468840dd88e3178ac4a9d41894d235015d9a9869498d8fa3c10410ec735/merged major:0 minor:427 fsType:overlay blockSize:0} overlay_0-432:{mountpoint:/var/lib/containers/storage/overlay/1d067b8a801e855abf930621b524dbf668bb4b4b0f8816497f31bca8b181d6c9/merged major:0 minor:432 fsType:overlay blockSize:0} overlay_0-437:{mountpoint:/var/lib/containers/storage/overlay/f1a44bccd9a813723fea4473d49e2ad68418efc9b6616fe5a91f72b46efc561a/merged major:0 minor:437 fsType:overlay blockSize:0} overlay_0-443:{mountpoint:/var/lib/containers/storage/overlay/f1e4d82c389c7e0125daa4e164fb24eeceeeebe437e071a7eb96152328b73a80/merged major:0 minor:443 fsType:overlay blockSize:0} overlay_0-465:{mountpoint:/var/lib/containers/storage/overlay/43fd82d45881bb279fed01ab1e99b455ebe84be5eb48cd04bc51964695e45f3c/merged major:0 minor:465 fsType:overlay blockSize:0} overlay_0-476:{mountpoint:/var/lib/containers/storage/overlay/922d03f1121303c33a89ab5fdd4635403359e8ffd3e14e78bea555c8f504fdc7/merged major:0 minor:476 fsType:overlay blockSize:0} overlay_0-478:{mountpoint:/var/lib/containers/storage/overlay/3a7b068a221806e4d5f1be51a1b9f6f361d8d387474cf99c775d93875b190f2d/merged major:0 minor:478 fsType:overlay blockSize:0} overlay_0-480:{mountpoint:/var/lib/containers/storage/overlay/d3b6742fd73a6d9e46b9efa592631124a61e4535e01fb1ea3b4f5f3819a41149/merged major:0 minor:480 fsType:overlay blockSize:0} overlay_0-482:{mountpoint:/var/lib/containers/storage/overlay/f0b539adceb0be1875d4b3e94388ea3f09f2fd81d18aa4c24600c917d5808622/merged major:0 minor:482 fsType:overlay blockSize:0} overlay_0-484:{mountpoint:/var/lib/containers/storage/overlay/582587a6d1ebb0a7b536d4e0cb31503376008cef447f48e494cc197b2db29726/merged major:0 minor:484 fsType:overlay blockSize:0} overlay_0-490:{mountpoint:/var/lib/containers/storage/overlay/48741c05ee658e6a17a0af2502c04960f4b764a4afc2895615318c8bfd84e33b/merged major:0 minor:490 fsType:overlay blockSize:0} overlay_0-495:{mountpoint:/var/lib/containers/storage/overlay/11e76f420c6f5297c479bc5b99e93aca511426fd2016b3ae7c66d8c5aeff5d33/merged major:0 minor:495 fsType:overlay blockSize:0} overlay_0-500:{mountpoint:/var/lib/containers/storage/overlay/57e39a61386975dc11ab1c741114dcf6440f2f7d9cc019c58144adf2192d551e/merged major:0 minor:500 fsType:overlay blockSize:0} overlay_0-505:{mountpoint:/var/lib/containers/storage/overlay/6ba644320f6d1b38ba24c9a9fc33e42ae6b0b4f747063aea6266075adf92f397/merged major:0 minor:505 fsType:overlay blockSize:0} overlay_0-510:{mountpoint:/var/lib/containers/storage/overlay/4541fd9d341888771d390306900d73aa9d1c896d9247c1bb8abb960c34f85fed/merged major:0 minor:510 fsType:overlay blockSize:0} overlay_0-515:{mountpoint:/var/lib/containers/storage/overlay/7d3e89a96bccb0c86ca8a2a214531cd6f131ebba3d19a0dd9978b6fc2b9e7fba/merged major:0 minor:515 fsType:overlay blockSize:0} overlay_0-521:{mountpoint:/var/lib/containers/storage/overlay/834002f06e1c9780f68b1e570c13561048f7c3e5e41df3e8bcc579b3f8063ea2/merged major:0 minor:521 fsType:overlay blockSize:0} overlay_0-526:{mountpoint:/var/lib/containers/storage/overlay/a1375557546dce3fb3008c53fcf13be88205dfcfb31b2398001986bb0e97b939/merged major:0 minor:526 fsType:overlay blockSize:0} overlay_0-65:{mountpoint:/var/lib/containers/storage/overlay/32887a4fecc16e78afdca011608efc7cbdf93e53692af23485647bba1e9cb088/merged major:0 minor:65 fsType:overlay blockSize:0} overlay_0-67:{mountpoint:/var/lib/containers/storage/overlay/c9f58a02780392ace5e7c29961daec8fe61fee79a43c697fa5148a754cd795a7/merged major:0 minor:67 fsType:overlay blockSize:0} overlay_0-69:{mountpoint:/var/lib/containers/storage/overlay/6c40a962102de899a6ab632b7b0ba4dab132684bbecbc8a5cfeef90c28fa1c95/merged major:0 minor:69 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/dc2da7a363c342f01eb54592bb961c8a59771f083734ee8134d05857fd827ec1/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/906713f112ffaf987bbd205fe899435b4e3baf4f3facc5c1de5f5e316b0d542b/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/e03ff6eb3b257fb105c8a75049374f2eee2eeeace7d202bdf08ff266e5d968f3/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/cd6802a909d1a24897273f05c92924f3e3855a1a2a42e61c2f643a8795263328/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/c3cba5de0b3f964141482b748d7ef691dd2f89ac440863e8752d1af50eafe68f/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/a6c0c807ee0f6a049bc0a58bcf672af509eaaadb73dfffd4d8b3949460e270df/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/caaa56be77ac97291c525a669f4a8573769bee9004c267e31cb52a62248a6e48/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 04 03:17:54.796927 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.796468 5953 manager.go:217] Machine: {Timestamp:2026-11-04 03:17:54.795744006 +0000 UTC m=+0.070374762 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3089592 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24c41d1b2460027f6ac682d0506f92 SystemUUID:ec24c41d-1b24-6002-7f6a-c682d0506f92 BootID:0243aa05-44b8-47e3-9575-cd1be2423faf Filesystems:[{Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1a6cc4f32dbf89779adc13674239ca74ae1dd88dd77759d62708826570f184b4/userdata/shm DeviceMajor:0 DeviceMinor:204 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/16659cac-4c32-4a94-90aa-8be0c6be33b2/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:231 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d6ebb345-a666-4ece-9afe-c1be0a44867b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:312 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271/volumes/kubernetes.io~projected/kube-api-access-tbcgk DeviceMajor:0 DeviceMinor:343 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-404 DeviceMajor:0 DeviceMinor:404 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1aa3f04c-b684-4426-8d46-a36d8c4e77be/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:467 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab/volumes/kubernetes.io~projected/kube-api-access-ngsgw DeviceMajor:0 DeviceMinor:53 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-67 DeviceMajor:0 DeviceMinor:67 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/93b3cbc9035ec7cec6ac37617c10933127bb5177af65587c6898c97cb381c6cd/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/44f960f8-1c9c-4a1b-a422-5477a6d49316/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:288 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6132bb22558201eafc6fe42434f09c31763011a7226766084223d4783a699f8f/userdata/shm DeviceMajor:0 DeviceMinor:304 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-65 DeviceMajor:0 DeviceMinor:65 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ba74b6466c9db3a2e9b4811b48d639a47993d10088c68fda03f7053e5d099e10/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:284 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e5e3933f-6610-46a7-934a-8ce205da98d9/volumes/kubernetes.io~projected/kube-api-access-zmxc6 DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2b9d93ac2db03e8dfc74e8608d2ceaeec92c843309e60201ee4057bc5fa939bb/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d6ebb345-a666-4ece-9afe-c1be0a44867b/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:317 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-379 DeviceMajor:0 DeviceMinor:379 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-437 DeviceMajor:0 DeviceMinor:437 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:455 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-505 DeviceMajor:0 DeviceMinor:505 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-184 DeviceMajor:0 DeviceMinor:184 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/57272665-8b2d-4854-bcb6-98f662716f4c/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:232 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:284 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1da1db8c80e0d30a8d1ff45a275e72414ff80b95ee8e6dba298e60a9807c7740/userdata/shm DeviceMajor:0 DeviceMinor:472 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-515 DeviceMajor:0 DeviceMinor:515 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-521 DeviceMajor:0 DeviceMinor:521 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5661b48df391450ec18633edf94716bde74a07933d0ef030521ad754a4909bf6/userdata/shm DeviceMajor:0 DeviceMinor:252 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d6ebb345-a666-4ece-9afe-c1be0a44867b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:316 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:454 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-495 DeviceMajor:0 DeviceMinor:495 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-500 DeviceMajor:0 DeviceMinor:500 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:341 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-384 DeviceMajor:0 DeviceMinor:384 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:285 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:458 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:461 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ca254b76-1d53-44aa-81e5-e40a81265145/volumes/kubernetes.io~projected/kube-api-access-fqc8z DeviceMajor:0 DeviceMinor:54 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/923f8598-9df4-46bb-8da0-73442c97522f/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c777436c764c665c4bb019ddad6db5c86ce03725b182ddcb23c951590f8595af/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-416 DeviceMajor:0 DeviceMinor:416 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-427 DeviceMajor:0 DeviceMinor:427 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/791796dc50b8bec54529e842614d63f8768af87659c325aa9ada29014e330c61/userdata/shm DeviceMajor:0 DeviceMinor:463 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/326a1269-c427-4470-86a6-70ee348c8791/volumes/kubernetes.io~projected/kube-api-access-8qzd8 DeviceMajor:0 DeviceMinor:52 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ca5887c7-01d6-44ee-ac30-8b4374663d3e/volumes/kubernetes.io~projected/kube-api-access-q24vc DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ea615393cda142a2d94cdcc77c191ac97a95d132f96c4c9a6636513c52ea39f3/userdata/shm DeviceMajor:0 DeviceMinor:346 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-465 DeviceMajor:0 DeviceMinor:465 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-510 DeviceMajor:0 DeviceMinor:510 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9f7d1536-9b78-4f13-ae4e-349db78d213e/volumes/kubernetes.io~projected/kube-api-access-kcw55 DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bd778358e255e28aa0c96fbbfcca8600ba0832bf0e6a2c63ac455194cecfc0a8/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9acbca43-6baa-4f8f-9eee-eb08dc731588/volumes/kubernetes.io~projected/kube-api-access-s869g DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4ab83526-361f-41de-b480-79fc45d7373a/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:354 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-432 DeviceMajor:0 DeviceMinor:432 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/75a2b016aaf3f6408b28d110698ecd053b2ae2ecbaa8aa7bcb68e1e6c2c30068/userdata/shm DeviceMajor:0 DeviceMinor:473 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:452 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-130 DeviceMajor:0 DeviceMinor:130 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/16659cac-4c32-4a94-90aa-8be0c6be33b2/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:233 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:279 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cd03c8ce-341a-4afe-93a1-2280392dd4f8/volumes/kubernetes.io~projected/kube-api-access-gtnzk DeviceMajor:0 DeviceMinor:339 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-387 DeviceMajor:0 DeviceMinor:387 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1846dbcff1a501528f398dfde9d918e157bceba20e37ce363fa71726ac064d1e/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f99e820db2e951c4e4a712acb3cd6c6d622e86bc03be5ed241a895818fb6e00c/userdata/shm DeviceMajor:0 DeviceMinor:245 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-476 DeviceMajor:0 DeviceMinor:476 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-478 DeviceMajor:0 DeviceMinor:478 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/16659cac-4c32-4a94-90aa-8be0c6be33b2/volumes/kubernetes.io~projected/kube-api-access-fcpzg DeviceMajor:0 DeviceMinor:235 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:283 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1aa3f04c-b684-4426-8d46-a36d8c4e77be/volumes/kubernetes.io~projected/kube-api-access-zdrr8 DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:285 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:290 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d6ebb345-a666-4ece-9afe-c1be0a44867b/volumes/kubernetes.io~projected/kube-api-access-5mr7j DeviceMajor:0 DeviceMinor:324 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4ab83526-361f-41de-b480-79fc45d7373a/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:355 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-480 DeviceMajor:0 DeviceMinor:480 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:452 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a040e5d8-7316-4a37-a045-9eb7c2eaf607/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:229 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d6ebb345-a666-4ece-9afe-c1be0a44867b/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:314 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d6ebb345-a666-4ece-9afe-c1be0a44867b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:315 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:340 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:342 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4ab83526-361f-41de-b480-79fc45d7373a/volumes/kubernetes.io~projected/kube-api-access-94vqb DeviceMajor:0 DeviceMinor:356 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:459 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6ec7ae50a4a7d2295b6890493085aaf6410dcfdc92d906ba9d10043571c9812b/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/57272665-8b2d-4854-bcb6-98f662716f4c/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:234 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4ab83526-361f-41de-b480-79fc45d7373a/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:353 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/63559e597407dae564a620e85c94cbf6318bd09f5c75313d0e6fdd9fba2f126a/userdata/shm DeviceMajor:0 DeviceMinor:470 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fb15c957-99e8-4c15-b382-788025661809/volumes/kubernetes.io~projected/kube-api-access-cdtwq DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c63fa297-f5dd-4c87-aa7b-f691ad61cb45/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c63fa297-f5dd-4c87-aa7b-f691ad61cb45/volumes/kubernetes.io~projected/kube-api-access-j2ftw DeviceMajor:0 DeviceMinor:208 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/719911b7-0c11-4c79-bb86-fdc043d9f9ab/volumes/kubernetes.io~projected/kube-api-access-n7xg7 DeviceMajor:0 DeviceMinor:469 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-135 DeviceMajor:0 DeviceMinor:135 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a54ba8507a55dea18a87dbe7af55c0f565972c855b2bf5b1b97864bc36355c1d/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ca5887c7-01d6-44ee-ac30-8b4374663d3e/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:50 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/44f960f8-1c9c-4a1b-a422-5477a6d49316/volumes/kubernetes.io~projected/kube-api-access-szld4 DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a040e5d8-7316-4a37-a045-9eb7c2eaf607/volumes/kubernetes.io~projected/kube-api-access-pvm8t DeviceMajor:0 DeviceMinor:230 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:289 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/64803c7435bf8b87b711ba3436cb3778c5f7bebc554c6d345f97132b5a0c8104/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a040e5d8-7316-4a37-a045-9eb7c2eaf607/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:225 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:453 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-490 DeviceMajor:0 DeviceMinor:490 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-403 DeviceMajor:0 DeviceMinor:403 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:456 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:460 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-484 DeviceMajor:0 DeviceMinor:484 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-526 DeviceMajor:0 DeviceMinor:526 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/57272665-8b2d-4854-bcb6-98f662716f4c/volumes/kubernetes.io~projected/kube-api-access-s7c4r DeviceMajor:0 DeviceMinor:236 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cd03c8ce-341a-4afe-93a1-2280392dd4f8/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:338 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:452 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/79ce3567247d25712b2d4037523f0437b4da86f3d90d0bac8859b00d5cf3eeee/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9acbca43-6baa-4f8f-9eee-eb08dc731588/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d6ebb345-a666-4ece-9afe-c1be0a44867b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:313 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1f357050039df0c66f46c3927903c9f913ef0340757c5043ead5991a62a0963b/userdata/shm DeviceMajor:0 DeviceMinor:344 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:284 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-386 DeviceMajor:0 DeviceMinor:386 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/988dc3031df7bc31a97c6a25b8e24cda782cef441f820edcda4057223ffb7655/userdata/shm DeviceMajor:0 DeviceMinor:64 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-179 DeviceMajor:0 DeviceMinor:179 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4ab83526-361f-41de-b480-79fc45d7373a/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:352 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/834b5587-1820-4089-a925-ab992fe654b4/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:468 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:287 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1f6b477e-a0fd-49c8-896d-c268a6c06d3f/volumes/kubernetes.io~projected/kube-api-access-vjwjp DeviceMajor:0 DeviceMinor:51 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:457 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-482 DeviceMajor:0 DeviceMinor:482 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/326a1269-c427-4470-86a6-70ee348c8791/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volumes/kubernetes.io~projected/kube-api-access-f99pm DeviceMajor:0 DeviceMinor:291 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/760c6e388680b0e6510ec7d509e014d206711baf48b8309f94bb523169a34ede/userdata/shm DeviceMajor:0 DeviceMinor:329 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-69 DeviceMajor:0 DeviceMinor:69 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:284 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-418 DeviceMajor:0 DeviceMinor:418 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ca5887c7-01d6-44ee-ac30-8b4374663d3e/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:286 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:448 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:452 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volumes/kubernetes.io~projected/kube-api-access-9rmmf DeviceMajor:0 DeviceMinor:462 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/326a1269-c427-4470-86a6-70ee348c8791/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:49 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-148 DeviceMajor:0 DeviceMinor:148 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d5c3b5befed935528d1a226b03c5e57dd4e4f5bee495cdf947cc098a3657f302/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c4998297ab19ab972cd9dbc2637cfe3b110924f917064937d89ed0ffc0959fbd/userdata/shm DeviceMajor:0 DeviceMinor:357 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-443 DeviceMajor:0 DeviceMinor:443 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d334d5ffb746244dc6e3456097b41613e54e96dfd4dbbb4e77995ab0fd416185/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1a6cc4f32dbf897 MacAddress:36:c6:cf:91:11:40 Speed:10000 Mtu:8901} {Name:1da1db8c80e0d30 MacAddress:2e:44:dc:d2:1f:94 Speed:10000 Mtu:8901} {Name:1f357050039df0c MacAddress:c6:ff:f5:da:51:e7 Speed:10000 Mtu:8901} {Name:2b9d93ac2db03e8 MacAddress:ca:7e:68:6e:51:a3 Speed:10000 Mtu:8901} {Name:6132bb22558201e MacAddress:4e:7d:3f:d2:86:97 Speed:10000 Mtu:8901} {Name:63559e597407dae MacAddress:ce:a1:b6:f0:5d:af Speed:10000 Mtu:8901} {Name:75a2b016aaf3f64 MacAddress:0e:61:9f:89:a8:f0 Speed:10000 Mtu:8901} {Name:760c6e388680b0e MacAddress:16:95:10:f4:84:d5 Speed:10000 Mtu:8901} {Name:791796dc50b8bec MacAddress:c2:bb:7b:bf:db:f9 Speed:10000 Mtu:8901} {Name:a54ba8507a55dea MacAddress:e2:de:fd:9f:57:ec Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:30:22:f9:d6:4d Speed:0 Mtu:9001} {Name:br-int MacAddress:6e:a9:33:a9:03:ab Speed:0 Mtu:8901} {Name:c4998297ab19ab9 MacAddress:8a:74:0a:82:5f:cf Speed:10000 Mtu:8901} {Name:d5c3b5befed9355 MacAddress:de:92:8f:99:b8:6d Speed:10000 Mtu:8901} {Name:ea615393cda142a MacAddress:62:2b:55:90:d3:81 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:30:22:f9:d6:4d Speed:0 Mtu:9001} {Name:f99e820db2e951c MacAddress:6e:a7:70:7e:fe:99 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:aa:52:d7:c5:d2:0c Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:0e:4f:b3:2b:63:e7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 04 03:17:54.797301 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.796932 5953 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 04 03:17:54.797301 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.796996 5953 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 04 03:17:54.797653 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.797577 5953 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 04 03:17:54.797937 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.797654 5953 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-6.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 04 03:17:54.798030 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.797941 5953 topology_manager.go:143] "Creating topology manager with none policy" Nov 04 03:17:54.798030 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.797956 5953 container_manager_linux.go:311] "Creating device plugin manager" Nov 04 03:17:54.798030 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.797976 5953 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 03:17:54.798030 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.797998 5953 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 03:17:54.798189 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.798144 5953 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 04 03:17:54.798393 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.798377 5953 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 04 03:17:54.798554 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.798542 5953 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 04 03:17:54.798652 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.798642 5953 kubelet.go:493] "Attempting to sync node with API server" Nov 04 03:17:54.798704 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.798665 5953 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 04 03:17:54.798704 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.798681 5953 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 04 03:17:54.798704 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.798699 5953 kubelet.go:405] "Adding apiserver pod source" Nov 04 03:17:54.798822 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.798741 5953 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 04 03:17:54.799689 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.799670 5953 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 04 03:17:54.799773 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.799690 5953 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 03:17:54.800750 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.800547 5953 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 04 03:17:54.803907 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.803883 5953 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 04 03:17:54.803907 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.803904 5953 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 03:17:54.804043 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.804013 5953 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 04 03:17:54.804223 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.804209 5953 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 04 03:17:54.804294 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.804240 5953 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 04 03:17:54.804357 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.804349 5953 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 04 03:17:54.804406 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.804362 5953 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 04 03:17:54.804406 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.804368 5953 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 04 03:17:54.804406 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.804373 5953 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 04 03:17:54.804406 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.804384 5953 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 04 03:17:54.804406 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.804393 5953 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 04 03:17:54.804406 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.804407 5953 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 04 03:17:54.804684 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.804413 5953 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 04 03:17:54.804684 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.804420 5953 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 04 03:17:54.804684 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.804427 5953 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 04 03:17:54.804684 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.804440 5953 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 04 03:17:54.804684 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.804450 5953 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 04 03:17:54.804684 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.804484 5953 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 04 03:17:54.804684 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.804493 5953 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 04 03:17:54.807984 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.807968 5953 server.go:1263] "Started kubelet" Nov 04 03:17:54.809201 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.808064 5953 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 04 03:17:54.809201 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.808651 5953 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 04 03:17:54.809201 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.808732 5953 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 04 03:17:54.809201 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.809128 5953 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 04 03:17:54.809433 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.809396 5953 server.go:317] "Adding debug handlers to kubelet server" Nov 04 03:17:54.810613 ip-10-0-133-6 systemd[1]: Started Kubernetes Kubelet. Nov 04 03:17:54.818663 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.814824 5953 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 04 03:17:54.818663 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.815053 5953 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 04 03:17:54.818663 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.815126 5953 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 03:11:43 +0000 UTC" deadline="2028-04-15 14:58:23.640390163 +0000 UTC" Nov 04 03:17:54.818663 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.815165 5953 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12683h40m28.825228629s" Nov 04 03:17:54.818663 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.815221 5953 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 04 03:17:54.818663 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.815229 5953 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 04 03:17:54.818663 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:17:54.815348 5953 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-6.us-west-1.compute.internal\" not found" Nov 04 03:17:54.818663 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.815643 5953 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 04 03:17:54.818663 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.817286 5953 factory.go:55] Registering systemd factory Nov 04 03:17:54.818663 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.817302 5953 factory.go:223] Registration of the systemd container factory successfully Nov 04 03:17:54.819364 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.819330 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" volumeName="kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-trusted-ca-bundle" seLinuxMountContext="" Nov 04 03:17:54.819433 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.819367 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca5887c7-01d6-44ee-ac30-8b4374663d3e" volumeName="kubernetes.io/empty-dir/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-tuned" seLinuxMountContext="" Nov 04 03:17:54.819433 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.819391 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16659cac-4c32-4a94-90aa-8be0c6be33b2" volumeName="kubernetes.io/configmap/16659cac-4c32-4a94-90aa-8be0c6be33b2-metrics-client-ca" seLinuxMountContext="" Nov 04 03:17:54.819433 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.819404 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16659cac-4c32-4a94-90aa-8be0c6be33b2" volumeName="kubernetes.io/empty-dir/16659cac-4c32-4a94-90aa-8be0c6be33b2-node-exporter-textfile" seLinuxMountContext="" Nov 04 03:17:54.819433 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.819414 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1aa3f04c-b684-4426-8d46-a36d8c4e77be" volumeName="kubernetes.io/secret/1aa3f04c-b684-4426-8d46-a36d8c4e77be-metrics-certs" seLinuxMountContext="" Nov 04 03:17:54.819433 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.819422 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c63fa297-f5dd-4c87-aa7b-f691ad61cb45" volumeName="kubernetes.io/projected/c63fa297-f5dd-4c87-aa7b-f691ad61cb45-kube-api-access-j2ftw" seLinuxMountContext="" Nov 04 03:17:54.819433 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.819434 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c63fa297-f5dd-4c87-aa7b-f691ad61cb45" volumeName="kubernetes.io/secret/c63fa297-f5dd-4c87-aa7b-f691ad61cb45-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 04 03:17:54.819718 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.819447 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" volumeName="kubernetes.io/empty-dir/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-config-out" seLinuxMountContext="" Nov 04 03:17:54.819718 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.819468 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" volumeName="kubernetes.io/projected/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-tls-assets" seLinuxMountContext="" Nov 04 03:17:54.819718 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.819484 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" volumeName="kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-kube-rbac-proxy" seLinuxMountContext="" Nov 04 03:17:54.819718 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.819493 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ab83526-361f-41de-b480-79fc45d7373a" volumeName="kubernetes.io/secret/4ab83526-361f-41de-b480-79fc45d7373a-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Nov 04 03:17:54.819718 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.819503 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" volumeName="kubernetes.io/empty-dir/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-alertmanager-main-db" seLinuxMountContext="" Nov 04 03:17:54.819718 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.819509 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5e3933f-6610-46a7-934a-8ce205da98d9" volumeName="kubernetes.io/projected/e5e3933f-6610-46a7-934a-8ce205da98d9-kube-api-access-zmxc6" seLinuxMountContext="" Nov 04 03:17:54.819718 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.819520 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271" volumeName="kubernetes.io/empty-dir/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-audit-log" seLinuxMountContext="" Nov 04 03:17:54.819718 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.819536 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57272665-8b2d-4854-bcb6-98f662716f4c" volumeName="kubernetes.io/projected/57272665-8b2d-4854-bcb6-98f662716f4c-kube-api-access-s7c4r" seLinuxMountContext="" Nov 04 03:17:54.819718 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.819562 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca254b76-1d53-44aa-81e5-e40a81265145" volumeName="kubernetes.io/configmap/ca254b76-1d53-44aa-81e5-e40a81265145-cni-binary-copy" seLinuxMountContext="" Nov 04 03:17:54.819718 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.819585 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="326a1269-c427-4470-86a6-70ee348c8791" volumeName="kubernetes.io/projected/326a1269-c427-4470-86a6-70ee348c8791-kube-api-access-8qzd8" seLinuxMountContext="" Nov 04 03:17:54.819718 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.819598 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57272665-8b2d-4854-bcb6-98f662716f4c" volumeName="kubernetes.io/configmap/57272665-8b2d-4854-bcb6-98f662716f4c-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 04 03:17:54.819718 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.819613 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271" volumeName="kubernetes.io/configmap/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-metrics-server-audit-profiles" seLinuxMountContext="" Nov 04 03:17:54.819718 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.819675 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca254b76-1d53-44aa-81e5-e40a81265145" volumeName="kubernetes.io/projected/ca254b76-1d53-44aa-81e5-e40a81265145-kube-api-access-fqc8z" seLinuxMountContext="" Nov 04 03:17:54.819718 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.819685 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6ebb345-a666-4ece-9afe-c1be0a44867b" volumeName="kubernetes.io/secret/d6ebb345-a666-4ece-9afe-c1be0a44867b-secret-thanos-querier-tls" seLinuxMountContext="" Nov 04 03:17:54.819718 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.819703 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" volumeName="kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-web-config" seLinuxMountContext="" Nov 04 03:17:54.819718 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.819724 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" volumeName="kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-prometheus-k8s-tls" seLinuxMountContext="" Nov 04 03:17:54.820464 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.819736 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="719911b7-0c11-4c79-bb86-fdc043d9f9ab" volumeName="kubernetes.io/projected/719911b7-0c11-4c79-bb86-fdc043d9f9ab-kube-api-access-n7xg7" seLinuxMountContext="" Nov 04 03:17:54.820464 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.819746 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271" volumeName="kubernetes.io/projected/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-kube-api-access-tbcgk" seLinuxMountContext="" Nov 04 03:17:54.820464 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.819763 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c63fa297-f5dd-4c87-aa7b-f691ad61cb45" volumeName="kubernetes.io/configmap/c63fa297-f5dd-4c87-aa7b-f691ad61cb45-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 04 03:17:54.820464 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.819783 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" volumeName="kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-main-tls" seLinuxMountContext="" Nov 04 03:17:54.820464 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.819814 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" volumeName="kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Nov 04 03:17:54.820464 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.819839 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16659cac-4c32-4a94-90aa-8be0c6be33b2" volumeName="kubernetes.io/secret/16659cac-4c32-4a94-90aa-8be0c6be33b2-node-exporter-tls" seLinuxMountContext="" Nov 04 03:17:54.820464 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.819855 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44f960f8-1c9c-4a1b-a422-5477a6d49316" volumeName="kubernetes.io/secret/44f960f8-1c9c-4a1b-a422-5477a6d49316-cert" seLinuxMountContext="" Nov 04 03:17:54.820464 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.819867 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" volumeName="kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Nov 04 03:17:54.820464 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.819895 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" volumeName="kubernetes.io/empty-dir/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-config-out" seLinuxMountContext="" Nov 04 03:17:54.820464 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.819907 5953 factory.go:153] Registering CRI-O factory Nov 04 03:17:54.820464 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.819909 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" volumeName="kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-metrics-client-certs" seLinuxMountContext="" Nov 04 03:17:54.820464 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.819930 5953 factory.go:223] Registration of the crio container factory successfully Nov 04 03:17:54.820464 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.819936 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f6b477e-a0fd-49c8-896d-c268a6c06d3f" volumeName="kubernetes.io/projected/1f6b477e-a0fd-49c8-896d-c268a6c06d3f-kube-api-access-vjwjp" seLinuxMountContext="" Nov 04 03:17:54.820464 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.819974 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9acbca43-6baa-4f8f-9eee-eb08dc731588" volumeName="kubernetes.io/empty-dir/9acbca43-6baa-4f8f-9eee-eb08dc731588-tmp-dir" seLinuxMountContext="" Nov 04 03:17:54.820464 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.819988 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1aa3f04c-b684-4426-8d46-a36d8c4e77be" volumeName="kubernetes.io/projected/1aa3f04c-b684-4426-8d46-a36d8c4e77be-kube-api-access-zdrr8" seLinuxMountContext="" Nov 04 03:17:54.820464 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.820009 5953 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 04 03:17:54.820464 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.820017 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ab83526-361f-41de-b480-79fc45d7373a" volumeName="kubernetes.io/secret/4ab83526-361f-41de-b480-79fc45d7373a-secret-telemeter-client" seLinuxMountContext="" Nov 04 03:17:54.820464 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.820033 5953 factory.go:103] Registering Raw factory Nov 04 03:17:54.820464 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.820044 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca5887c7-01d6-44ee-ac30-8b4374663d3e" volumeName="kubernetes.io/projected/ca5887c7-01d6-44ee-ac30-8b4374663d3e-kube-api-access-q24vc" seLinuxMountContext="" Nov 04 03:17:54.820464 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.820057 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd03c8ce-341a-4afe-93a1-2280392dd4f8" volumeName="kubernetes.io/secret/cd03c8ce-341a-4afe-93a1-2280392dd4f8-monitoring-plugin-cert" seLinuxMountContext="" Nov 04 03:17:54.820464 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.820111 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6ebb345-a666-4ece-9afe-c1be0a44867b" volumeName="kubernetes.io/configmap/d6ebb345-a666-4ece-9afe-c1be0a44867b-metrics-client-ca" seLinuxMountContext="" Nov 04 03:17:54.820464 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.820129 5953 manager.go:1199] Started watching for new ooms in manager Nov 04 03:17:54.820464 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.820131 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb15c957-99e8-4c15-b382-788025661809" volumeName="kubernetes.io/empty-dir/fb15c957-99e8-4c15-b382-788025661809-tmp-dir" seLinuxMountContext="" Nov 04 03:17:54.820464 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.820224 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab" volumeName="kubernetes.io/configmap/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-cni-sysctl-allowlist" seLinuxMountContext="" Nov 04 03:17:54.820464 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.820244 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f7d1536-9b78-4f13-ae4e-349db78d213e" volumeName="kubernetes.io/configmap/9f7d1536-9b78-4f13-ae4e-349db78d213e-iptables-alerter-script" seLinuxMountContext="" Nov 04 03:17:54.820464 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.820267 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271" volumeName="kubernetes.io/secret/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-secret-metrics-server-client-certs" seLinuxMountContext="" Nov 04 03:17:54.820464 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.820284 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" volumeName="kubernetes.io/empty-dir/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-db" seLinuxMountContext="" Nov 04 03:17:54.820464 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.820304 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a040e5d8-7316-4a37-a045-9eb7c2eaf607" volumeName="kubernetes.io/secret/a040e5d8-7316-4a37-a045-9eb7c2eaf607-openshift-state-metrics-tls" seLinuxMountContext="" Nov 04 03:17:54.820464 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.820323 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="923f8598-9df4-46bb-8da0-73442c97522f" volumeName="kubernetes.io/configmap/923f8598-9df4-46bb-8da0-73442c97522f-konnectivity-ca" seLinuxMountContext="" Nov 04 03:17:54.820464 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.820340 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab" volumeName="kubernetes.io/configmap/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-cni-binary-copy" seLinuxMountContext="" Nov 04 03:17:54.820464 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.820359 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c63fa297-f5dd-4c87-aa7b-f691ad61cb45" volumeName="kubernetes.io/empty-dir/c63fa297-f5dd-4c87-aa7b-f691ad61cb45-data-volume" seLinuxMountContext="" Nov 04 03:17:54.820464 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.820375 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a040e5d8-7316-4a37-a045-9eb7c2eaf607" volumeName="kubernetes.io/configmap/a040e5d8-7316-4a37-a045-9eb7c2eaf607-metrics-client-ca" seLinuxMountContext="" Nov 04 03:17:54.820464 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.820413 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9acbca43-6baa-4f8f-9eee-eb08dc731588" volumeName="kubernetes.io/projected/9acbca43-6baa-4f8f-9eee-eb08dc731588-kube-api-access-s869g" seLinuxMountContext="" Nov 04 03:17:54.820464 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.820439 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9acbca43-6baa-4f8f-9eee-eb08dc731588" volumeName="kubernetes.io/secret/9acbca43-6baa-4f8f-9eee-eb08dc731588-metrics-tls" seLinuxMountContext="" Nov 04 03:17:54.820464 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.820458 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd03c8ce-341a-4afe-93a1-2280392dd4f8" volumeName="kubernetes.io/projected/cd03c8ce-341a-4afe-93a1-2280392dd4f8-kube-api-access-gtnzk" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.820650 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" volumeName="kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.820691 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" volumeName="kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-thanos-prometheus-http-client-file" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.820709 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" volumeName="kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.820755 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57272665-8b2d-4854-bcb6-98f662716f4c" volumeName="kubernetes.io/secret/57272665-8b2d-4854-bcb6-98f662716f4c-kube-state-metrics-tls" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.820783 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6ebb345-a666-4ece-9afe-c1be0a44867b" volumeName="kubernetes.io/projected/d6ebb345-a666-4ece-9afe-c1be0a44867b-kube-api-access-5mr7j" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.820796 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6ebb345-a666-4ece-9afe-c1be0a44867b" volumeName="kubernetes.io/secret/d6ebb345-a666-4ece-9afe-c1be0a44867b-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.820813 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ab83526-361f-41de-b480-79fc45d7373a" volumeName="kubernetes.io/secret/4ab83526-361f-41de-b480-79fc45d7373a-telemeter-client-tls" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.820838 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57272665-8b2d-4854-bcb6-98f662716f4c" volumeName="kubernetes.io/configmap/57272665-8b2d-4854-bcb6-98f662716f4c-metrics-client-ca" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.820850 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" volumeName="kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-config-volume" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.820870 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" volumeName="kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.820925 5953 manager.go:319] Starting recovery of all containers Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.820944 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb15c957-99e8-4c15-b382-788025661809" volumeName="kubernetes.io/projected/fb15c957-99e8-4c15-b382-788025661809-kube-api-access-cdtwq" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.821094 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44f960f8-1c9c-4a1b-a422-5477a6d49316" volumeName="kubernetes.io/projected/44f960f8-1c9c-4a1b-a422-5477a6d49316-kube-api-access-szld4" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.821115 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6ebb345-a666-4ece-9afe-c1be0a44867b" volumeName="kubernetes.io/empty-dir/d6ebb345-a666-4ece-9afe-c1be0a44867b-tmp" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.821129 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16659cac-4c32-4a94-90aa-8be0c6be33b2" volumeName="kubernetes.io/secret/16659cac-4c32-4a94-90aa-8be0c6be33b2-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.821163 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ab83526-361f-41de-b480-79fc45d7373a" volumeName="kubernetes.io/projected/4ab83526-361f-41de-b480-79fc45d7373a-kube-api-access-94vqb" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.821379 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57272665-8b2d-4854-bcb6-98f662716f4c" volumeName="kubernetes.io/empty-dir/57272665-8b2d-4854-bcb6-98f662716f4c-volume-directive-shadow" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.821399 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" volumeName="kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-web-config" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.821420 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" volumeName="kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-grpc-tls" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.821436 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a040e5d8-7316-4a37-a045-9eb7c2eaf607" volumeName="kubernetes.io/projected/a040e5d8-7316-4a37-a045-9eb7c2eaf607-kube-api-access-pvm8t" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.821462 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271" volumeName="kubernetes.io/secret/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-client-ca-bundle" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.821474 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ab83526-361f-41de-b480-79fc45d7373a" volumeName="kubernetes.io/configmap/4ab83526-361f-41de-b480-79fc45d7373a-serving-certs-ca-bundle" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.821506 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ab83526-361f-41de-b480-79fc45d7373a" volumeName="kubernetes.io/configmap/4ab83526-361f-41de-b480-79fc45d7373a-metrics-client-ca" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.821520 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ab83526-361f-41de-b480-79fc45d7373a" volumeName="kubernetes.io/secret/4ab83526-361f-41de-b480-79fc45d7373a-federate-client-tls" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.821545 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab" volumeName="kubernetes.io/projected/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-kube-api-access-ngsgw" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.821558 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="326a1269-c427-4470-86a6-70ee348c8791" volumeName="kubernetes.io/configmap/326a1269-c427-4470-86a6-70ee348c8791-env-overrides" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.821570 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="326a1269-c427-4470-86a6-70ee348c8791" volumeName="kubernetes.io/configmap/326a1269-c427-4470-86a6-70ee348c8791-ovnkube-script-lib" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.821597 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9acbca43-6baa-4f8f-9eee-eb08dc731588" volumeName="kubernetes.io/configmap/9acbca43-6baa-4f8f-9eee-eb08dc731588-config-volume" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.821642 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271" volumeName="kubernetes.io/secret/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-secret-metrics-server-tls" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.821660 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" volumeName="kubernetes.io/projected/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-kube-api-access-f99pm" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.821744 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" volumeName="kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.821756 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" volumeName="kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.821947 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16659cac-4c32-4a94-90aa-8be0c6be33b2" volumeName="kubernetes.io/configmap/16659cac-4c32-4a94-90aa-8be0c6be33b2-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.821976 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ab83526-361f-41de-b480-79fc45d7373a" volumeName="kubernetes.io/configmap/4ab83526-361f-41de-b480-79fc45d7373a-telemeter-trusted-ca-bundle" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.821997 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6ebb345-a666-4ece-9afe-c1be0a44867b" volumeName="kubernetes.io/secret/d6ebb345-a666-4ece-9afe-c1be0a44867b-secret-grpc-tls" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.822010 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16659cac-4c32-4a94-90aa-8be0c6be33b2" volumeName="kubernetes.io/projected/16659cac-4c32-4a94-90aa-8be0c6be33b2-kube-api-access-fcpzg" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.822032 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f6b477e-a0fd-49c8-896d-c268a6c06d3f" volumeName="kubernetes.io/configmap/1f6b477e-a0fd-49c8-896d-c268a6c06d3f-serviceca" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.822046 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6ebb345-a666-4ece-9afe-c1be0a44867b" volumeName="kubernetes.io/secret/d6ebb345-a666-4ece-9afe-c1be0a44867b-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.822060 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca5887c7-01d6-44ee-ac30-8b4374663d3e" volumeName="kubernetes.io/empty-dir/ca5887c7-01d6-44ee-ac30-8b4374663d3e-tmp" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.822099 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6ebb345-a666-4ece-9afe-c1be0a44867b" volumeName="kubernetes.io/secret/d6ebb345-a666-4ece-9afe-c1be0a44867b-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Nov 04 03:17:54.822165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.822186 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="326a1269-c427-4470-86a6-70ee348c8791" volumeName="kubernetes.io/configmap/326a1269-c427-4470-86a6-70ee348c8791-ovnkube-config" seLinuxMountContext="" Nov 04 03:17:54.825354 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.822208 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57272665-8b2d-4854-bcb6-98f662716f4c" volumeName="kubernetes.io/secret/57272665-8b2d-4854-bcb6-98f662716f4c-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 04 03:17:54.825354 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.822236 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" volumeName="kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-cluster-tls-config" seLinuxMountContext="" Nov 04 03:17:54.825354 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.822280 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" volumeName="kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Nov 04 03:17:54.825354 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.822294 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" volumeName="kubernetes.io/projected/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-kube-api-access-9rmmf" seLinuxMountContext="" Nov 04 03:17:54.825354 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.822323 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" volumeName="kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-config" seLinuxMountContext="" Nov 04 03:17:54.825354 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.822355 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f7d1536-9b78-4f13-ae4e-349db78d213e" volumeName="kubernetes.io/projected/9f7d1536-9b78-4f13-ae4e-349db78d213e-kube-api-access-kcw55" seLinuxMountContext="" Nov 04 03:17:54.825354 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.822370 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a040e5d8-7316-4a37-a045-9eb7c2eaf607" volumeName="kubernetes.io/secret/a040e5d8-7316-4a37-a045-9eb7c2eaf607-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 04 03:17:54.825354 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.822592 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="834b5587-1820-4089-a925-ab992fe654b4" volumeName="kubernetes.io/secret/834b5587-1820-4089-a925-ab992fe654b4-original-pull-secret" seLinuxMountContext="" Nov 04 03:17:54.825354 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.822641 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" volumeName="kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-configmap-metrics-client-ca" seLinuxMountContext="" Nov 04 03:17:54.825354 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.822661 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="923f8598-9df4-46bb-8da0-73442c97522f" volumeName="kubernetes.io/secret/923f8598-9df4-46bb-8da0-73442c97522f-agent-certs" seLinuxMountContext="" Nov 04 03:17:54.825354 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.822681 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab" volumeName="kubernetes.io/configmap/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 04 03:17:54.825354 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.822696 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca254b76-1d53-44aa-81e5-e40a81265145" volumeName="kubernetes.io/configmap/ca254b76-1d53-44aa-81e5-e40a81265145-multus-daemon-config" seLinuxMountContext="" Nov 04 03:17:54.825354 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.822716 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6ebb345-a666-4ece-9afe-c1be0a44867b" volumeName="kubernetes.io/secret/d6ebb345-a666-4ece-9afe-c1be0a44867b-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Nov 04 03:17:54.825354 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.822730 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" volumeName="kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Nov 04 03:17:54.825354 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.822768 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271" volumeName="kubernetes.io/configmap/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 04 03:17:54.825354 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.822784 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" volumeName="kubernetes.io/configmap/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Nov 04 03:17:54.825354 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.822802 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" volumeName="kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Nov 04 03:17:54.825354 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.822848 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" volumeName="kubernetes.io/projected/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-tls-assets" seLinuxMountContext="" Nov 04 03:17:54.825354 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.822862 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="326a1269-c427-4470-86a6-70ee348c8791" volumeName="kubernetes.io/secret/326a1269-c427-4470-86a6-70ee348c8791-ovn-node-metrics-cert" seLinuxMountContext="" Nov 04 03:17:54.825354 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.822954 5953 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" volumeName="kubernetes.io/configmap/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-metrics-client-ca" seLinuxMountContext="" Nov 04 03:17:54.825354 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.822973 5953 reconstruct.go:97] "Volume reconstruction finished" Nov 04 03:17:54.825354 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.822983 5953 reconciler.go:29] "Reconciler: start to sync state" Nov 04 03:17:54.845514 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.845396 5953 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 04 03:17:54.851373 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.848199 5953 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 04 03:17:54.851373 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.848225 5953 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 04 03:17:54.851373 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.848255 5953 kubelet.go:2521] "Starting kubelet main sync loop" Nov 04 03:17:54.851373 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:17:54.848293 5953 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 04 03:17:54.865308 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.865269 5953 generic.go:356] "Generic (PLEG): container finished" podID="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" containerID="c5b0e0618d7907ea3905d068f6441be77893e0bdeb245dd8dab7618ad3fc1818" exitCode=0 Nov 04 03:17:54.878064 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.878033 5953 generic.go:356] "Generic (PLEG): container finished" podID="16659cac-4c32-4a94-90aa-8be0c6be33b2" containerID="7c17d08aee32ed9811d81e7bc47f2635f8bc415a63cf65ecafc82dc3f89f2367" exitCode=0 Nov 04 03:17:54.890819 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.890784 5953 generic.go:356] "Generic (PLEG): container finished" podID="96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab" containerID="85f1612483fabbfd5b5da78ef0902893c1986baf04d13b0f3e6824419956f71f" exitCode=0 Nov 04 03:17:54.890819 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.890817 5953 generic.go:356] "Generic (PLEG): container finished" podID="96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab" containerID="f715a0a5876fb149627a0ba95b1ec978568f4d5dc3ac8a8b3e005db64aeb7906" exitCode=0 Nov 04 03:17:54.890819 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.890828 5953 generic.go:356] "Generic (PLEG): container finished" podID="96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab" containerID="77575ea23166825ba6f7ec42146d03fc1b15d469ef0a09fb2740de06ce2d1339" exitCode=0 Nov 04 03:17:54.891096 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.890836 5953 generic.go:356] "Generic (PLEG): container finished" podID="96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab" containerID="9a372e85dccd7c6e891a11f276d1f9e6113c43a01ded557ad82da989d7b395fd" exitCode=0 Nov 04 03:17:54.891096 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.890841 5953 generic.go:356] "Generic (PLEG): container finished" podID="96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab" containerID="6721ba231dc26a949079274cd0723294ef0a0eac51544cb55964e4bd6aeae9d1" exitCode=0 Nov 04 03:17:54.891096 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.890847 5953 generic.go:356] "Generic (PLEG): container finished" podID="96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab" containerID="06ba6bb8082f6aac9dd2851112ce7b3220b92fb2c0dc5f7cceef6a8dc5cf81e1" exitCode=0 Nov 04 03:17:54.901468 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.901439 5953 generic.go:356] "Generic (PLEG): container finished" podID="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" containerID="f4ac97bd1d0e01b9f860c3ffeb1a6073e2aa15efea50981c2053c645efdbe607" exitCode=0 Nov 04 03:17:54.908932 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.908899 5953 generic.go:356] "Generic (PLEG): container finished" podID="12436f4d0565699f03041c74b80c999a" containerID="9198cc3bcfb59628049f7b350ac2652eda5b2d8873955670a914423b2d0f33be" exitCode=0 Nov 04 03:17:54.915481 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:17:54.915418 5953 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-6.us-west-1.compute.internal\" not found" Nov 04 03:17:54.948396 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:17:54.948353 5953 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 04 03:17:54.955593 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.955572 5953 manager.go:324] Recovery completed Nov 04 03:17:54.981174 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.981143 5953 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 03:17:54.983100 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.983080 5953 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-6.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 03:17:54.983234 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.983112 5953 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-6.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 03:17:54.983234 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.983122 5953 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-6.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 03:17:54.986490 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.986472 5953 cpu_manager.go:226] "Starting" policy="none" Nov 04 03:17:54.986490 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.986487 5953 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 04 03:17:54.986598 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.986494 5953 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 03:17:54.986598 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.986507 5953 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 04 03:17:54.986684 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.986660 5953 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 04 03:17:54.986684 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.986672 5953 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 04 03:17:54.986755 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.986690 5953 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 04 03:17:54.986755 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.986696 5953 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 04 03:17:54.986755 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.986702 5953 policy_none.go:50] "Start" Nov 04 03:17:54.986755 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.986749 5953 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 04 03:17:54.986861 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.986759 5953 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 04 03:17:54.986972 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.986954 5953 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 04 03:17:54.986972 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.986967 5953 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 04 03:17:54.987026 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.986975 5953 policy_none.go:44] "Start" Nov 04 03:17:54.987542 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.987522 5953 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-6.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 03:17:54.987660 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.987558 5953 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-6.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 03:17:54.987660 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.987570 5953 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-6.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 03:17:54.992023 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.991995 5953 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 04 03:17:54.994583 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.994566 5953 manager.go:341] "Starting Device Plugin manager" Nov 04 03:17:54.994686 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:17:54.994592 5953 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 04 03:17:54.994686 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.994601 5953 server.go:87] "Starting device plugin registration server" Nov 04 03:17:54.994952 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.994938 5953 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 04 03:17:54.995016 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.994951 5953 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 04 03:17:54.995069 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.995023 5953 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 04 03:17:54.995156 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.995119 5953 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 04 03:17:54.995210 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.995172 5953 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 04 03:17:54.995210 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.995180 5953 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 04 03:17:54.995341 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.995244 5953 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-04T03:17:54.995154409Z","UUID":"e1e32f96-e629-40d0-862c-ada0d6dd6a2f","Handler":null,"Name":"","Endpoint":""} Nov 04 03:17:54.996938 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.996920 5953 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 04 03:17:54.997029 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:54.996947 5953 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 04 03:17:54.997852 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:17:54.997830 5953 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 04 03:17:54.998084 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:17:54.997874 5953 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-6.us-west-1.compute.internal\" not found" Nov 04 03:17:55.011838 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.011814 5953 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 03:17:55.011838 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.011828 5953 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 03:17:55.012033 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.011989 5953 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 04 03:17:55.012890 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.012875 5953 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 03:17:55.095793 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.095737 5953 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 03:17:55.096984 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.096963 5953 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-6.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 03:17:55.097100 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.097003 5953 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-6.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 03:17:55.097100 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.097018 5953 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-6.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 03:17:55.097100 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.097086 5953 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:17:55.112137 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.112108 5953 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 03:17:55.149200 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.149123 5953 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-6.us-west-1.compute.internal"] Nov 04 03:17:55.149911 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.149894 5953 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:17:55.150019 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.149878 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-6.us-west-1.compute.internal" event={"ID":"ca990c51a9c4be7e8fb143b232789941","Type":"ContainerStarted","Data":"c20a8c1167238ad5e0d857c02273fad6f50a3bab6739b88d986191be44efc5e7"} Nov 04 03:17:55.150076 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.150043 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-6.us-west-1.compute.internal" event={"ID":"ca990c51a9c4be7e8fb143b232789941","Type":"ContainerStarted","Data":"79ce3567247d25712b2d4037523f0437b4da86f3d90d0bac8859b00d5cf3eeee"} Nov 04 03:17:55.150107 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.150085 5953 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:17:55.150136 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.150112 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal" event={"ID":"12436f4d0565699f03041c74b80c999a","Type":"ContainerStarted","Data":"bcd00165b2b4556e7338124d93b06f75dfd4a739ed25aaa4a8fa96d029a03393"} Nov 04 03:17:55.150165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.150132 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal" event={"ID":"12436f4d0565699f03041c74b80c999a","Type":"ContainerDied","Data":"9198cc3bcfb59628049f7b350ac2652eda5b2d8873955670a914423b2d0f33be"} Nov 04 03:17:55.150165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.150152 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal" event={"ID":"12436f4d0565699f03041c74b80c999a","Type":"ContainerStarted","Data":"d334d5ffb746244dc6e3456097b41613e54e96dfd4dbbb4e77995ab0fd416185"} Nov 04 03:17:55.212929 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.212856 5953 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 03:17:55.212929 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:17:55.212917 5953 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-6.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:17:55.213438 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.213421 5953 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 03:17:55.213478 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:17:55.213462 5953 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:17:55.223303 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.223282 5953 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:17:55.223400 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.223349 5953 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:17:55.313060 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.313024 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ca990c51a9c4be7e8fb143b232789941-config\") pod \"kube-apiserver-proxy-ip-10-0-133-6.us-west-1.compute.internal\" (UID: \"ca990c51a9c4be7e8fb143b232789941\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:17:55.313243 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.313074 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/12436f4d0565699f03041c74b80c999a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal\" (UID: \"12436f4d0565699f03041c74b80c999a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:17:55.313243 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.313092 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12436f4d0565699f03041c74b80c999a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal\" (UID: \"12436f4d0565699f03041c74b80c999a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:17:55.414063 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.414027 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/12436f4d0565699f03041c74b80c999a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal\" (UID: \"12436f4d0565699f03041c74b80c999a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:17:55.414236 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.414062 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/12436f4d0565699f03041c74b80c999a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal\" (UID: \"12436f4d0565699f03041c74b80c999a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:17:55.414236 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.414098 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12436f4d0565699f03041c74b80c999a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal\" (UID: \"12436f4d0565699f03041c74b80c999a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:17:55.414236 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.414125 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ca990c51a9c4be7e8fb143b232789941-config\") pod \"kube-apiserver-proxy-ip-10-0-133-6.us-west-1.compute.internal\" (UID: \"ca990c51a9c4be7e8fb143b232789941\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:17:55.414236 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.414172 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ca990c51a9c4be7e8fb143b232789941-config\") pod \"kube-apiserver-proxy-ip-10-0-133-6.us-west-1.compute.internal\" (UID: \"ca990c51a9c4be7e8fb143b232789941\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:17:55.414236 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.414185 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12436f4d0565699f03041c74b80c999a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal\" (UID: \"12436f4d0565699f03041c74b80c999a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:17:55.800294 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.800260 5953 apiserver.go:52] "Watching apiserver" Nov 04 03:17:55.906170 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.906131 5953 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 04 03:17:55.906965 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.906936 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-jf99x","openshift-multus/network-metrics-daemon-dzg2c","kube-system/konnectivity-agent-pmv6n","openshift-dns/node-resolver-r9xhm","openshift-monitoring/node-exporter-7c9tl","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb","kube-system/global-pull-secret-syncer-8rmjv","openshift-monitoring/prometheus-k8s-0","openshift-network-operator/iptables-alerter-s2zcf","openshift-ovn-kubernetes/ovnkube-node-bqm4b","openshift-ingress-canary/ingress-canary-wv6zh","openshift-cluster-node-tuning-operator/tuned-7qqx5","openshift-dns/dns-default-jg2pw","openshift-insights/insights-runtime-extractor-p8x4k","openshift-monitoring/alertmanager-main-0","openshift-monitoring/thanos-querier-7c4874f885-hqvmr","openshift-multus/multus-t2twg","openshift-image-registry/node-ca-wj6dr","openshift-multus/multus-additional-cni-plugins-q5p48","kube-system/kube-apiserver-proxy-ip-10-0-133-6.us-west-1.compute.internal","openshift-monitoring/monitoring-plugin-77f4f76594-kf7dm","openshift-monitoring/telemeter-client-745f9d6695-rlnsq","openshift-monitoring/metrics-server-748ff79954-pr5lc","openshift-network-diagnostics/network-check-target-2hjpn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal","openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc"] Nov 04 03:17:55.926467 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.926441 5953 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:17:55.926648 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.926553 5953 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:17:55.961983 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.961952 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 04 03:17:55.962162 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.961954 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 04 03:17:55.962162 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.962016 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 04 03:17:55.962162 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.961986 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 04 03:17:55.962162 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.962090 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 04 03:17:55.962162 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.962094 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 04 03:17:55.962162 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.962122 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qhl58\"" Nov 04 03:17:55.962162 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.962149 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 04 03:17:55.962162 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.962158 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 04 03:17:55.962550 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.962183 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 04 03:17:55.962550 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.962195 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5pzz6\"" Nov 04 03:17:55.962550 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.962183 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 04 03:17:55.962550 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.962228 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 04 03:17:55.962550 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.962239 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-f24jt\"" Nov 04 03:17:55.962550 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.962241 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 04 03:17:55.962550 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.962328 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 04 03:17:55.962550 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.962400 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9c548\"" Nov 04 03:17:55.962550 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.962481 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 04 03:17:55.962550 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.962478 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5q5qr\"" Nov 04 03:17:55.962550 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.962533 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 04 03:17:55.962550 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.962544 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 04 03:17:55.962957 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.962580 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-8nk47\"" Nov 04 03:17:55.963073 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.963060 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 04 03:17:55.963173 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.963162 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-kfskp\"" Nov 04 03:17:55.965845 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.965832 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7jn7l\"" Nov 04 03:17:55.966231 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.966220 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 04 03:17:55.966543 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.966526 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 04 03:17:55.966782 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.966768 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 04 03:17:55.966849 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.966784 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x9qdq\"" Nov 04 03:17:55.966849 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.966803 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 04 03:17:55.966849 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.966843 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 04 03:17:55.966984 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.966845 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 04 03:17:55.966984 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.966912 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 04 03:17:55.966984 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.966957 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 04 03:17:55.967160 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.967147 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 04 03:17:55.967294 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.967273 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ztc7x\"" Nov 04 03:17:55.967462 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.967445 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-cj8mj\"" Nov 04 03:17:55.967532 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.967490 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 04 03:17:55.967532 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.967510 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 04 03:17:55.967646 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.967545 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 04 03:17:55.967764 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.967743 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 04 03:17:55.967862 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.967789 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n89nd\"" Nov 04 03:17:55.967862 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.967846 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 04 03:17:55.968023 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.968008 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 04 03:17:55.968127 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.968084 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-s72nt\"" Nov 04 03:17:55.968530 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.968510 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 04 03:17:55.968593 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:55.968554 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 04 03:17:56.013420 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.013385 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-2qrjk\"" Nov 04 03:17:56.013593 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.013430 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 04 03:17:56.013593 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.013439 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 04 03:17:56.013593 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.013448 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 04 03:17:56.013593 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.013477 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-sslkp\"" Nov 04 03:17:56.013893 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.013875 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 04 03:17:56.013956 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.013884 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 04 03:17:56.013956 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.013934 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 04 03:17:56.014062 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.014047 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 04 03:17:56.014120 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.014103 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 04 03:17:56.014264 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.014249 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 04 03:17:56.014830 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.014813 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 04 03:17:56.014901 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.014840 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 04 03:17:56.014901 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.014849 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 04 03:17:56.014901 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.014852 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 04 03:17:56.015022 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.015008 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 04 03:17:56.016072 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.016054 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c63fa297-f5dd-4c87-aa7b-f691ad61cb45-crio-socket\") pod \"insights-runtime-extractor-p8x4k\" (UID: \"c63fa297-f5dd-4c87-aa7b-f691ad61cb45\") " pod="openshift-insights/insights-runtime-extractor-p8x4k" Nov 04 03:17:56.016140 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.016091 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca5887c7-01d6-44ee-ac30-8b4374663d3e-tmp\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:17:56.016140 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.016121 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-kubelet\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.016234 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.016183 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-log-socket\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.016234 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.016218 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca5887c7-01d6-44ee-ac30-8b4374663d3e-tmp\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:17:56.016294 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.016230 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9f7d1536-9b78-4f13-ae4e-349db78d213e-host-slash\") pod \"iptables-alerter-s2zcf\" (UID: \"9f7d1536-9b78-4f13-ae4e-349db78d213e\") " pod="openshift-network-operator/iptables-alerter-s2zcf" Nov 04 03:17:56.016294 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.016269 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-system-cni-dir\") pod \"multus-additional-cni-plugins-q5p48\" (UID: \"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab\") " pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:17:56.016294 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.016290 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fb15c957-99e8-4c15-b382-788025661809-hosts-file\") pod \"node-resolver-r9xhm\" (UID: \"fb15c957-99e8-4c15-b382-788025661809\") " pod="openshift-dns/node-resolver-r9xhm" Nov 04 03:17:56.016376 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.016333 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7xg7\" (UniqueName: \"kubernetes.io/projected/719911b7-0c11-4c79-bb86-fdc043d9f9ab-kube-api-access-n7xg7\") pod \"network-check-target-2hjpn\" (UID: \"719911b7-0c11-4c79-bb86-fdc043d9f9ab\") " pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:17:56.016405 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.016384 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-cni-netd\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.016513 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.016496 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9acbca43-6baa-4f8f-9eee-eb08dc731588-metrics-tls\") pod \"dns-default-jg2pw\" (UID: \"9acbca43-6baa-4f8f-9eee-eb08dc731588\") " pod="openshift-dns/dns-default-jg2pw" Nov 04 03:17:56.016547 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.016529 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9acbca43-6baa-4f8f-9eee-eb08dc731588-tmp-dir\") pod \"dns-default-jg2pw\" (UID: \"9acbca43-6baa-4f8f-9eee-eb08dc731588\") " pod="openshift-dns/dns-default-jg2pw" Nov 04 03:17:56.016601 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.016585 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q5p48\" (UID: \"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab\") " pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:17:56.016678 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.016660 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngsgw\" (UniqueName: \"kubernetes.io/projected/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-kube-api-access-ngsgw\") pod \"multus-additional-cni-plugins-q5p48\" (UID: \"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab\") " pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:17:56.016712 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.016678 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9acbca43-6baa-4f8f-9eee-eb08dc731588-tmp-dir\") pod \"dns-default-jg2pw\" (UID: \"9acbca43-6baa-4f8f-9eee-eb08dc731588\") " pod="openshift-dns/dns-default-jg2pw" Nov 04 03:17:56.016748 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.016717 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e5e3933f-6610-46a7-934a-8ce205da98d9-sys-fs\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:17:56.016794 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.016778 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9acbca43-6baa-4f8f-9eee-eb08dc731588-metrics-tls\") pod \"dns-default-jg2pw\" (UID: \"9acbca43-6baa-4f8f-9eee-eb08dc731588\") " pod="openshift-dns/dns-default-jg2pw" Nov 04 03:17:56.016832 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.016789 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q5p48\" (UID: \"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab\") " pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:17:56.016865 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.016827 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-host\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:17:56.017100 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.017079 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-tuned\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:17:56.017165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.017152 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-tuned\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:17:56.017250 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.017235 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qzd8\" (UniqueName: \"kubernetes.io/projected/326a1269-c427-4470-86a6-70ee348c8791-kube-api-access-8qzd8\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.017288 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.017271 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca254b76-1d53-44aa-81e5-e40a81265145-cni-binary-copy\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.017325 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.017295 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-host-var-lib-cni-multus\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.017325 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.017314 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2ftw\" (UniqueName: \"kubernetes.io/projected/c63fa297-f5dd-4c87-aa7b-f691ad61cb45-kube-api-access-j2ftw\") pod \"insights-runtime-extractor-p8x4k\" (UID: \"c63fa297-f5dd-4c87-aa7b-f691ad61cb45\") " pod="openshift-insights/insights-runtime-extractor-p8x4k" Nov 04 03:17:56.017390 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.017333 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-system-cni-dir\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.017390 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.017371 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-host-run-netns\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.017452 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.017396 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-multus-conf-dir\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.017452 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.017425 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqc8z\" (UniqueName: \"kubernetes.io/projected/ca254b76-1d53-44aa-81e5-e40a81265145-kube-api-access-fqc8z\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.017521 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.017478 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e5e3933f-6610-46a7-934a-8ce205da98d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:17:56.017557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.017517 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca254b76-1d53-44aa-81e5-e40a81265145-cni-binary-copy\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.017647 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.017597 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aa3f04c-b684-4426-8d46-a36d8c4e77be-metrics-certs\") pod \"network-metrics-daemon-dzg2c\" (UID: \"1aa3f04c-b684-4426-8d46-a36d8c4e77be\") " pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:17:56.017733 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.017716 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c63fa297-f5dd-4c87-aa7b-f691ad61cb45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p8x4k\" (UID: \"c63fa297-f5dd-4c87-aa7b-f691ad61cb45\") " pod="openshift-insights/insights-runtime-extractor-p8x4k" Nov 04 03:17:56.017783 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.017766 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-systemd\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:17:56.017819 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.017791 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aa3f04c-b684-4426-8d46-a36d8c4e77be-metrics-certs\") pod \"network-metrics-daemon-dzg2c\" (UID: \"1aa3f04c-b684-4426-8d46-a36d8c4e77be\") " pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:17:56.017854 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.017823 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-host-run-multus-certs\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.017882 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.017863 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-etc-kubernetes\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.017934 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.017921 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e5e3933f-6610-46a7-934a-8ce205da98d9-device-dir\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:17:56.017934 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.017927 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c63fa297-f5dd-4c87-aa7b-f691ad61cb45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p8x4k\" (UID: \"c63fa297-f5dd-4c87-aa7b-f691ad61cb45\") " pod="openshift-insights/insights-runtime-extractor-p8x4k" Nov 04 03:17:56.018012 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.017997 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-sys\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:17:56.018043 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.018026 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-lib-modules\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:17:56.018087 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.018051 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q24vc\" (UniqueName: \"kubernetes.io/projected/ca5887c7-01d6-44ee-ac30-8b4374663d3e-kube-api-access-q24vc\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:17:56.018087 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.018065 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-var-lib-openvswitch\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.018087 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.018081 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f6b477e-a0fd-49c8-896d-c268a6c06d3f-host\") pod \"node-ca-wj6dr\" (UID: \"1f6b477e-a0fd-49c8-896d-c268a6c06d3f\") " pod="openshift-image-registry/node-ca-wj6dr" Nov 04 03:17:56.018194 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.018173 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/834b5587-1820-4089-a925-ab992fe654b4-original-pull-secret\") pod \"global-pull-secret-syncer-8rmjv\" (UID: \"834b5587-1820-4089-a925-ab992fe654b4\") " pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:17:56.018224 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.018195 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-sysconfig\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:17:56.018266 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.018228 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-run\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:17:56.018301 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.018264 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-run-openvswitch\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.018301 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.018283 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-cni-bin\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.018364 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.018299 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-run-ovn-kubernetes\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.018364 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.018347 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.018420 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.018382 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/326a1269-c427-4470-86a6-70ee348c8791-ovnkube-script-lib\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.018420 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.018408 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1f6b477e-a0fd-49c8-896d-c268a6c06d3f-serviceca\") pod \"node-ca-wj6dr\" (UID: \"1f6b477e-a0fd-49c8-896d-c268a6c06d3f\") " pod="openshift-image-registry/node-ca-wj6dr" Nov 04 03:17:56.018500 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.018483 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/834b5587-1820-4089-a925-ab992fe654b4-original-pull-secret\") pod \"global-pull-secret-syncer-8rmjv\" (UID: \"834b5587-1820-4089-a925-ab992fe654b4\") " pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:17:56.018532 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.018491 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjwjp\" (UniqueName: \"kubernetes.io/projected/1f6b477e-a0fd-49c8-896d-c268a6c06d3f-kube-api-access-vjwjp\") pod \"node-ca-wj6dr\" (UID: \"1f6b477e-a0fd-49c8-896d-c268a6c06d3f\") " pod="openshift-image-registry/node-ca-wj6dr" Nov 04 03:17:56.018655 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.018643 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-multus-cni-dir\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.018688 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.018668 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-hostroot\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.018732 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.018682 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/326a1269-c427-4470-86a6-70ee348c8791-ovnkube-script-lib\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.018732 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.018701 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-cnibin\") pod \"multus-additional-cni-plugins-q5p48\" (UID: \"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab\") " pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:17:56.018787 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.018737 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb15c957-99e8-4c15-b382-788025661809-tmp-dir\") pod \"node-resolver-r9xhm\" (UID: \"fb15c957-99e8-4c15-b382-788025661809\") " pod="openshift-dns/node-resolver-r9xhm" Nov 04 03:17:56.018787 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.018776 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdtwq\" (UniqueName: \"kubernetes.io/projected/fb15c957-99e8-4c15-b382-788025661809-kube-api-access-cdtwq\") pod \"node-resolver-r9xhm\" (UID: \"fb15c957-99e8-4c15-b382-788025661809\") " pod="openshift-dns/node-resolver-r9xhm" Nov 04 03:17:56.018873 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.018857 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-node-log\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.018937 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.018905 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9acbca43-6baa-4f8f-9eee-eb08dc731588-config-volume\") pod \"dns-default-jg2pw\" (UID: \"9acbca43-6baa-4f8f-9eee-eb08dc731588\") " pod="openshift-dns/dns-default-jg2pw" Nov 04 03:17:56.018937 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.018906 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb15c957-99e8-4c15-b382-788025661809-tmp-dir\") pod \"node-resolver-r9xhm\" (UID: \"fb15c957-99e8-4c15-b382-788025661809\") " pod="openshift-dns/node-resolver-r9xhm" Nov 04 03:17:56.019096 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.018954 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s869g\" (UniqueName: \"kubernetes.io/projected/9acbca43-6baa-4f8f-9eee-eb08dc731588-kube-api-access-s869g\") pod \"dns-default-jg2pw\" (UID: \"9acbca43-6baa-4f8f-9eee-eb08dc731588\") " pod="openshift-dns/dns-default-jg2pw" Nov 04 03:17:56.019096 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.019005 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c63fa297-f5dd-4c87-aa7b-f691ad61cb45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p8x4k\" (UID: \"c63fa297-f5dd-4c87-aa7b-f691ad61cb45\") " pod="openshift-insights/insights-runtime-extractor-p8x4k" Nov 04 03:17:56.019096 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.019036 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-kubernetes\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:17:56.019211 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.019111 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9acbca43-6baa-4f8f-9eee-eb08dc731588-config-volume\") pod \"dns-default-jg2pw\" (UID: \"9acbca43-6baa-4f8f-9eee-eb08dc731588\") " pod="openshift-dns/dns-default-jg2pw" Nov 04 03:17:56.019211 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.019152 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-etc-openvswitch\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.019211 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.019179 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/326a1269-c427-4470-86a6-70ee348c8791-env-overrides\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.019211 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.019196 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1f6b477e-a0fd-49c8-896d-c268a6c06d3f-serviceca\") pod \"node-ca-wj6dr\" (UID: \"1f6b477e-a0fd-49c8-896d-c268a6c06d3f\") " pod="openshift-image-registry/node-ca-wj6dr" Nov 04 03:17:56.019329 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.019212 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/834b5587-1820-4089-a925-ab992fe654b4-dbus\") pod \"global-pull-secret-syncer-8rmjv\" (UID: \"834b5587-1820-4089-a925-ab992fe654b4\") " pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:17:56.019329 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.019241 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-multus-socket-dir-parent\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.019329 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.019217 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c63fa297-f5dd-4c87-aa7b-f691ad61cb45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p8x4k\" (UID: \"c63fa297-f5dd-4c87-aa7b-f691ad61cb45\") " pod="openshift-insights/insights-runtime-extractor-p8x4k" Nov 04 03:17:56.019408 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.019342 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-host-run-k8s-cni-cncf-io\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.019408 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.019346 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/326a1269-c427-4470-86a6-70ee348c8791-env-overrides\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.019408 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.019377 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-host-var-lib-kubelet\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.019489 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.019405 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q5p48\" (UID: \"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab\") " pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:17:56.019518 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.019498 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e5e3933f-6610-46a7-934a-8ce205da98d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:17:56.019518 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.019507 5953 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 04 03:17:56.019571 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.019517 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-run-netns\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.019571 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.019537 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9f7d1536-9b78-4f13-ae4e-349db78d213e-iptables-alerter-script\") pod \"iptables-alerter-s2zcf\" (UID: \"9f7d1536-9b78-4f13-ae4e-349db78d213e\") " pod="openshift-network-operator/iptables-alerter-s2zcf" Nov 04 03:17:56.019571 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.019554 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/923f8598-9df4-46bb-8da0-73442c97522f-konnectivity-ca\") pod \"konnectivity-agent-pmv6n\" (UID: \"923f8598-9df4-46bb-8da0-73442c97522f\") " pod="kube-system/konnectivity-agent-pmv6n" Nov 04 03:17:56.019571 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.019559 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q5p48\" (UID: \"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab\") " pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:17:56.019702 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.019593 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-cni-binary-copy\") pod \"multus-additional-cni-plugins-q5p48\" (UID: \"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab\") " pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:17:56.019759 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.019742 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-sysctl-d\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:17:56.019804 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.019758 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9f7d1536-9b78-4f13-ae4e-349db78d213e-iptables-alerter-script\") pod \"iptables-alerter-s2zcf\" (UID: \"9f7d1536-9b78-4f13-ae4e-349db78d213e\") " pod="openshift-network-operator/iptables-alerter-s2zcf" Nov 04 03:17:56.019804 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.019777 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/923f8598-9df4-46bb-8da0-73442c97522f-konnectivity-ca\") pod \"konnectivity-agent-pmv6n\" (UID: \"923f8598-9df4-46bb-8da0-73442c97522f\") " pod="kube-system/konnectivity-agent-pmv6n" Nov 04 03:17:56.019804 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.019789 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-sysctl-conf\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:17:56.019804 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.019794 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-cni-binary-copy\") pod \"multus-additional-cni-plugins-q5p48\" (UID: \"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab\") " pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:17:56.019913 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.019836 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcw55\" (UniqueName: \"kubernetes.io/projected/9f7d1536-9b78-4f13-ae4e-349db78d213e-kube-api-access-kcw55\") pod \"iptables-alerter-s2zcf\" (UID: \"9f7d1536-9b78-4f13-ae4e-349db78d213e\") " pod="openshift-network-operator/iptables-alerter-s2zcf" Nov 04 03:17:56.019913 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.019853 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-run-ovn\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.019913 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.019870 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/326a1269-c427-4470-86a6-70ee348c8791-ovn-node-metrics-cert\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.019913 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.019885 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/834b5587-1820-4089-a925-ab992fe654b4-kubelet-config\") pod \"global-pull-secret-syncer-8rmjv\" (UID: \"834b5587-1820-4089-a925-ab992fe654b4\") " pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:17:56.019913 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.019900 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44f960f8-1c9c-4a1b-a422-5477a6d49316-cert\") pod \"ingress-canary-wv6zh\" (UID: \"44f960f8-1c9c-4a1b-a422-5477a6d49316\") " pod="openshift-ingress-canary/ingress-canary-wv6zh" Nov 04 03:17:56.020073 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.019929 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q5p48\" (UID: \"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab\") " pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:17:56.020073 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.019950 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e5e3933f-6610-46a7-934a-8ce205da98d9-socket-dir\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:17:56.020073 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.019987 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmxc6\" (UniqueName: \"kubernetes.io/projected/e5e3933f-6610-46a7-934a-8ce205da98d9-kube-api-access-zmxc6\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:17:56.020073 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.020022 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c63fa297-f5dd-4c87-aa7b-f691ad61cb45-data-volume\") pod \"insights-runtime-extractor-p8x4k\" (UID: \"c63fa297-f5dd-4c87-aa7b-f691ad61cb45\") " pod="openshift-insights/insights-runtime-extractor-p8x4k" Nov 04 03:17:56.020073 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.020050 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-slash\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.020073 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.020064 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-host-var-lib-cni-bin\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.020358 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.020100 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/326a1269-c427-4470-86a6-70ee348c8791-ovn-node-metrics-cert\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.020358 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.020170 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-var-lib-kubelet\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:17:56.020358 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.020204 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-systemd-units\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.020358 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.020216 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c63fa297-f5dd-4c87-aa7b-f691ad61cb45-data-volume\") pod \"insights-runtime-extractor-p8x4k\" (UID: \"c63fa297-f5dd-4c87-aa7b-f691ad61cb45\") " pod="openshift-insights/insights-runtime-extractor-p8x4k" Nov 04 03:17:56.020358 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.020232 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/923f8598-9df4-46bb-8da0-73442c97522f-agent-certs\") pod \"konnectivity-agent-pmv6n\" (UID: \"923f8598-9df4-46bb-8da0-73442c97522f\") " pod="kube-system/konnectivity-agent-pmv6n" Nov 04 03:17:56.020358 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.020236 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44f960f8-1c9c-4a1b-a422-5477a6d49316-cert\") pod \"ingress-canary-wv6zh\" (UID: \"44f960f8-1c9c-4a1b-a422-5477a6d49316\") " pod="openshift-ingress-canary/ingress-canary-wv6zh" Nov 04 03:17:56.020358 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.020319 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szld4\" (UniqueName: \"kubernetes.io/projected/44f960f8-1c9c-4a1b-a422-5477a6d49316-kube-api-access-szld4\") pod \"ingress-canary-wv6zh\" (UID: \"44f960f8-1c9c-4a1b-a422-5477a6d49316\") " pod="openshift-ingress-canary/ingress-canary-wv6zh" Nov 04 03:17:56.020358 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.020352 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e5e3933f-6610-46a7-934a-8ce205da98d9-registration-dir\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:17:56.020749 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.020367 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-modprobe-d\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:17:56.020749 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.020384 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-run-systemd\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.020749 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.020424 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/326a1269-c427-4470-86a6-70ee348c8791-ovnkube-config\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.020749 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.020452 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-cnibin\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.020749 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.020463 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/923f8598-9df4-46bb-8da0-73442c97522f-agent-certs\") pod \"konnectivity-agent-pmv6n\" (UID: \"923f8598-9df4-46bb-8da0-73442c97522f\") " pod="kube-system/konnectivity-agent-pmv6n" Nov 04 03:17:56.020749 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.020481 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ca254b76-1d53-44aa-81e5-e40a81265145-multus-daemon-config\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.020749 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.020583 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/326a1269-c427-4470-86a6-70ee348c8791-ovnkube-config\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.020749 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.020597 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdrr8\" (UniqueName: \"kubernetes.io/projected/1aa3f04c-b684-4426-8d46-a36d8c4e77be-kube-api-access-zdrr8\") pod \"network-metrics-daemon-dzg2c\" (UID: \"1aa3f04c-b684-4426-8d46-a36d8c4e77be\") " pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:17:56.020749 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.020690 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ca254b76-1d53-44aa-81e5-e40a81265145-multus-daemon-config\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.026747 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.026727 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 04 03:17:56.026869 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.026854 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 04 03:17:56.027913 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.027893 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 04 03:17:56.029320 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.029290 5953 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 03:17:56.029393 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:17:56.029354 5953 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-6.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:17:56.029498 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.029479 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 04 03:17:56.029824 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.029808 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-c8m9w\"" Nov 04 03:17:56.029824 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.029817 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 04 03:17:56.030025 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.030012 5953 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 03:17:56.030093 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:17:56.030075 5953 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-6.us-west-1.compute.internal" Nov 04 03:17:56.030142 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.030127 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e4spppbfh1spp\"" Nov 04 03:17:56.030289 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.030273 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 04 03:17:56.030891 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.030875 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 04 03:17:56.031242 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.031226 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 04 03:17:56.031423 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.031409 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 04 03:17:56.031484 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.031436 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 04 03:17:56.031532 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.031520 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 04 03:17:56.032189 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.032171 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 04 03:17:56.063808 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.063736 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 04 03:17:56.063808 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.063733 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 04 03:17:56.064272 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.064246 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xmpr4\"" Nov 04 03:17:56.064447 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.064435 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 04 03:17:56.064497 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.064452 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 04 03:17:56.064497 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.064469 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 04 03:17:56.064497 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.064480 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 04 03:17:56.064497 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.064483 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 04 03:17:56.064799 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.064775 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 04 03:17:56.064799 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.064794 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 04 03:17:56.064972 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.064789 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 04 03:17:56.064972 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.064818 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 04 03:17:56.064972 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.064787 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-mwn26\"" Nov 04 03:17:56.064972 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.064830 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 04 03:17:56.064972 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.064845 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-tdnkm\"" Nov 04 03:17:56.064972 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.064979 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 04 03:17:56.065266 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.064990 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 04 03:17:56.065744 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.065501 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 04 03:17:56.065744 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.065511 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 04 03:17:56.065744 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.065503 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 04 03:17:56.065744 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.065529 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 04 03:17:56.065744 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.065566 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 04 03:17:56.065744 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.065568 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 04 03:17:56.065744 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.065535 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 04 03:17:56.065744 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.065592 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 04 03:17:56.065744 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.065582 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 04 03:17:56.065744 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.065580 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 04 03:17:56.065744 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.065592 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 04 03:17:56.065744 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.065658 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 04 03:17:56.065744 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.065664 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-krstf\"" Nov 04 03:17:56.065744 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.065674 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 04 03:17:56.065744 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.065678 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-tmmps\"" Nov 04 03:17:56.065744 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.065719 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 04 03:17:56.065744 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.065744 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-124bnl54asoc8\"" Nov 04 03:17:56.065744 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.065746 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 04 03:17:56.066312 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.065806 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5drsvv7kbdor6\"" Nov 04 03:17:56.066312 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.065903 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-bh6gw\"" Nov 04 03:17:56.066312 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.065943 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-sn5zj\"" Nov 04 03:17:56.066312 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.065961 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 04 03:17:56.066312 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.065999 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4fbdz\"" Nov 04 03:17:56.081655 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.081634 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjwjp\" (UniqueName: \"kubernetes.io/projected/1f6b477e-a0fd-49c8-896d-c268a6c06d3f-kube-api-access-vjwjp\") pod \"node-ca-wj6dr\" (UID: \"1f6b477e-a0fd-49c8-896d-c268a6c06d3f\") " pod="openshift-image-registry/node-ca-wj6dr" Nov 04 03:17:56.081773 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.081613 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s869g\" (UniqueName: \"kubernetes.io/projected/9acbca43-6baa-4f8f-9eee-eb08dc731588-kube-api-access-s869g\") pod \"dns-default-jg2pw\" (UID: \"9acbca43-6baa-4f8f-9eee-eb08dc731588\") " pod="openshift-dns/dns-default-jg2pw" Nov 04 03:17:56.081773 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.081738 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqc8z\" (UniqueName: \"kubernetes.io/projected/ca254b76-1d53-44aa-81e5-e40a81265145-kube-api-access-fqc8z\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.081773 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.081762 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngsgw\" (UniqueName: \"kubernetes.io/projected/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-kube-api-access-ngsgw\") pod \"multus-additional-cni-plugins-q5p48\" (UID: \"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab\") " pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:17:56.081773 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.081765 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcw55\" (UniqueName: \"kubernetes.io/projected/9f7d1536-9b78-4f13-ae4e-349db78d213e-kube-api-access-kcw55\") pod \"iptables-alerter-s2zcf\" (UID: \"9f7d1536-9b78-4f13-ae4e-349db78d213e\") " pod="openshift-network-operator/iptables-alerter-s2zcf" Nov 04 03:17:56.083502 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.083474 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q24vc\" (UniqueName: \"kubernetes.io/projected/ca5887c7-01d6-44ee-ac30-8b4374663d3e-kube-api-access-q24vc\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:17:56.083717 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.083702 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7xg7\" (UniqueName: \"kubernetes.io/projected/719911b7-0c11-4c79-bb86-fdc043d9f9ab-kube-api-access-n7xg7\") pod \"network-check-target-2hjpn\" (UID: \"719911b7-0c11-4c79-bb86-fdc043d9f9ab\") " pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:17:56.083799 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.083782 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdrr8\" (UniqueName: \"kubernetes.io/projected/1aa3f04c-b684-4426-8d46-a36d8c4e77be-kube-api-access-zdrr8\") pod \"network-metrics-daemon-dzg2c\" (UID: \"1aa3f04c-b684-4426-8d46-a36d8c4e77be\") " pod="openshift-multus/network-metrics-daemon-dzg2c" Nov 04 03:17:56.086302 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.086265 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdtwq\" (UniqueName: \"kubernetes.io/projected/fb15c957-99e8-4c15-b382-788025661809-kube-api-access-cdtwq\") pod \"node-resolver-r9xhm\" (UID: \"fb15c957-99e8-4c15-b382-788025661809\") " pod="openshift-dns/node-resolver-r9xhm" Nov 04 03:17:56.086776 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.086756 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmxc6\" (UniqueName: \"kubernetes.io/projected/e5e3933f-6610-46a7-934a-8ce205da98d9-kube-api-access-zmxc6\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:17:56.089371 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.089348 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qzd8\" (UniqueName: \"kubernetes.io/projected/326a1269-c427-4470-86a6-70ee348c8791-kube-api-access-8qzd8\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.089511 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.089491 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2ftw\" (UniqueName: \"kubernetes.io/projected/c63fa297-f5dd-4c87-aa7b-f691ad61cb45-kube-api-access-j2ftw\") pod \"insights-runtime-extractor-p8x4k\" (UID: \"c63fa297-f5dd-4c87-aa7b-f691ad61cb45\") " pod="openshift-insights/insights-runtime-extractor-p8x4k" Nov 04 03:17:56.089762 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.089744 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szld4\" (UniqueName: \"kubernetes.io/projected/44f960f8-1c9c-4a1b-a422-5477a6d49316-kube-api-access-szld4\") pod \"ingress-canary-wv6zh\" (UID: \"44f960f8-1c9c-4a1b-a422-5477a6d49316\") " pod="openshift-ingress-canary/ingress-canary-wv6zh" Nov 04 03:17:56.121521 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.121489 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-run-systemd\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.121708 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.121527 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-config-out\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:56.121708 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.121551 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-secret-metrics-server-client-certs\") pod \"metrics-server-748ff79954-pr5lc\" (UID: \"bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271\") " pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" Nov 04 03:17:56.121708 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.121573 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/16659cac-4c32-4a94-90aa-8be0c6be33b2-node-exporter-tls\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:56.121708 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.121593 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-cnibin\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.121708 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.121602 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-run-systemd\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.121708 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.121633 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c63fa297-f5dd-4c87-aa7b-f691ad61cb45-crio-socket\") pod \"insights-runtime-extractor-p8x4k\" (UID: \"c63fa297-f5dd-4c87-aa7b-f691ad61cb45\") " pod="openshift-insights/insights-runtime-extractor-p8x4k" Nov 04 03:17:56.121708 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.121684 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c63fa297-f5dd-4c87-aa7b-f691ad61cb45-crio-socket\") pod \"insights-runtime-extractor-p8x4k\" (UID: \"c63fa297-f5dd-4c87-aa7b-f691ad61cb45\") " pod="openshift-insights/insights-runtime-extractor-p8x4k" Nov 04 03:17:56.121708 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.121687 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-cnibin\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.121708 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.121690 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-kubelet\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.121708 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.121687 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-config-out\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:56.122178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.121718 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d6ebb345-a666-4ece-9afe-c1be0a44867b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:56.122178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.121736 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-system-cni-dir\") pod \"multus-additional-cni-plugins-q5p48\" (UID: \"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab\") " pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:17:56.122178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.121747 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-kubelet\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.122178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.121755 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fb15c957-99e8-4c15-b382-788025661809-hosts-file\") pod \"node-resolver-r9xhm\" (UID: \"fb15c957-99e8-4c15-b382-788025661809\") " pod="openshift-dns/node-resolver-r9xhm" Nov 04 03:17:56.122178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.121806 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.122178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.121803 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-system-cni-dir\") pod \"multus-additional-cni-plugins-q5p48\" (UID: \"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab\") " pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:17:56.122178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.121822 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.122178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.121845 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.122178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.121866 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fb15c957-99e8-4c15-b382-788025661809-hosts-file\") pod \"node-resolver-r9xhm\" (UID: \"fb15c957-99e8-4c15-b382-788025661809\") " pod="openshift-dns/node-resolver-r9xhm" Nov 04 03:17:56.122178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.121873 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e5e3933f-6610-46a7-934a-8ce205da98d9-sys-fs\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:17:56.122178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.121875 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/16659cac-4c32-4a94-90aa-8be0c6be33b2-node-exporter-tls\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:56.122178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.121897 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:56.122178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.121875 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-secret-metrics-server-client-certs\") pod \"metrics-server-748ff79954-pr5lc\" (UID: \"bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271\") " pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" Nov 04 03:17:56.122178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.121921 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.122178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.121926 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d6ebb345-a666-4ece-9afe-c1be0a44867b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:56.122178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.121948 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16659cac-4c32-4a94-90aa-8be0c6be33b2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:56.122178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.121967 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e5e3933f-6610-46a7-934a-8ce205da98d9-sys-fs\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:17:56.122178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.121974 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:56.122178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122002 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57272665-8b2d-4854-bcb6-98f662716f4c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-ztrlc\" (UID: \"57272665-8b2d-4854-bcb6-98f662716f4c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc" Nov 04 03:17:56.122178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122013 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.122178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122029 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d6ebb345-a666-4ece-9afe-c1be0a44867b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:56.122178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122036 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.122178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122043 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.122178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122058 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6ebb345-a666-4ece-9afe-c1be0a44867b-metrics-client-ca\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:56.122178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122089 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:56.122178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122110 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-system-cni-dir\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.122178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122164 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.122178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122174 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-system-cni-dir\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.122178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122181 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16659cac-4c32-4a94-90aa-8be0c6be33b2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122225 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e5e3933f-6610-46a7-934a-8ce205da98d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122236 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57272665-8b2d-4854-bcb6-98f662716f4c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-ztrlc\" (UID: \"57272665-8b2d-4854-bcb6-98f662716f4c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122247 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d6ebb345-a666-4ece-9afe-c1be0a44867b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122264 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-systemd\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122223 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122277 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6ebb345-a666-4ece-9afe-c1be0a44867b-metrics-client-ca\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122289 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e5e3933f-6610-46a7-934a-8ce205da98d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122283 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-sys\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122310 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-sys\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122313 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-var-lib-openvswitch\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122338 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122340 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-var-lib-openvswitch\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122362 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-host-run-multus-certs\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122356 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-systemd\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122378 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-etc-kubernetes\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122400 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-audit-log\") pod \"metrics-server-748ff79954-pr5lc\" (UID: \"bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271\") " pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122422 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/16659cac-4c32-4a94-90aa-8be0c6be33b2-node-exporter-textfile\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122424 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-host-run-multus-certs\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122431 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-etc-kubernetes\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122439 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-run\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122453 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-run-openvswitch\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122461 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122474 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-cni-bin\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122481 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-audit-log\") pod \"metrics-server-748ff79954-pr5lc\" (UID: \"bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271\") " pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122500 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-run\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122503 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/16659cac-4c32-4a94-90aa-8be0c6be33b2-node-exporter-textfile\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122501 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-run-openvswitch\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122511 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122533 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-cni-bin\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122547 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122650 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122668 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-secret-metrics-server-tls\") pod \"metrics-server-748ff79954-pr5lc\" (UID: \"bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271\") " pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122684 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-web-config\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122696 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122701 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-hostroot\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122728 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-node-log\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122753 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-748ff79954-pr5lc\" (UID: \"bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271\") " pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122775 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-hostroot\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122778 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-kubernetes\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122822 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-etc-openvswitch\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122850 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ab83526-361f-41de-b480-79fc45d7373a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122853 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-node-log\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122874 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d6ebb345-a666-4ece-9afe-c1be0a44867b-secret-grpc-tls\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122881 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-web-config\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122825 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-kubernetes\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122898 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-secret-metrics-server-tls\") pod \"metrics-server-748ff79954-pr5lc\" (UID: \"bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271\") " pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122914 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122926 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-etc-openvswitch\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122901 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mr7j\" (UniqueName: \"kubernetes.io/projected/d6ebb345-a666-4ece-9afe-c1be0a44867b-kube-api-access-5mr7j\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122962 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtnzk\" (UniqueName: \"kubernetes.io/projected/cd03c8ce-341a-4afe-93a1-2280392dd4f8-kube-api-access-gtnzk\") pod \"monitoring-plugin-77f4f76594-kf7dm\" (UID: \"cd03c8ce-341a-4afe-93a1-2280392dd4f8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-kf7dm" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122979 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/834b5587-1820-4089-a925-ab992fe654b4-dbus\") pod \"global-pull-secret-syncer-8rmjv\" (UID: \"834b5587-1820-4089-a925-ab992fe654b4\") " pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.122996 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-multus-socket-dir-parent\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123017 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-host-run-k8s-cni-cncf-io\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123036 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-748ff79954-pr5lc\" (UID: \"bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271\") " pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123042 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-run-netns\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123064 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123076 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ab83526-361f-41de-b480-79fc45d7373a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123086 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123097 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/834b5587-1820-4089-a925-ab992fe654b4-dbus\") pod \"global-pull-secret-syncer-8rmjv\" (UID: \"834b5587-1820-4089-a925-ab992fe654b4\") " pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123110 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a040e5d8-7316-4a37-a045-9eb7c2eaf607-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-jf99x\" (UID: \"a040e5d8-7316-4a37-a045-9eb7c2eaf607\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jf99x" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123152 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-host-run-k8s-cni-cncf-io\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123170 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d6ebb345-a666-4ece-9afe-c1be0a44867b-secret-grpc-tls\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123196 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-multus-socket-dir-parent\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123243 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-sysctl-conf\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123335 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123345 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a040e5d8-7316-4a37-a045-9eb7c2eaf607-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-jf99x\" (UID: \"a040e5d8-7316-4a37-a045-9eb7c2eaf607\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jf99x" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123366 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123392 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-run-netns\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123407 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123444 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ab83526-361f-41de-b480-79fc45d7373a-serving-certs-ca-bundle\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123471 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123477 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-sysctl-conf\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123476 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123508 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123535 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-run-ovn\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123560 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-web-config\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123589 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvm8t\" (UniqueName: \"kubernetes.io/projected/a040e5d8-7316-4a37-a045-9eb7c2eaf607-kube-api-access-pvm8t\") pod \"openshift-state-metrics-7bd7489957-jf99x\" (UID: \"a040e5d8-7316-4a37-a045-9eb7c2eaf607\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jf99x" Nov 04 03:17:56.123557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123601 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123639 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d6ebb345-a666-4ece-9afe-c1be0a44867b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123668 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q5p48\" (UID: \"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab\") " pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123591 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-run-ovn\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123681 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123694 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e5e3933f-6610-46a7-934a-8ce205da98d9-socket-dir\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123719 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123736 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q5p48\" (UID: \"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab\") " pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123742 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123670 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ab83526-361f-41de-b480-79fc45d7373a-serving-certs-ca-bundle\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123777 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-metrics-server-audit-profiles\") pod \"metrics-server-748ff79954-pr5lc\" (UID: \"bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271\") " pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123801 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-web-config\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123808 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ab83526-361f-41de-b480-79fc45d7373a-metrics-client-ca\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123822 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e5e3933f-6610-46a7-934a-8ce205da98d9-socket-dir\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123832 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e5e3933f-6610-46a7-934a-8ce205da98d9-registration-dir\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123863 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d6ebb345-a666-4ece-9afe-c1be0a44867b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123870 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123892 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e5e3933f-6610-46a7-934a-8ce205da98d9-registration-dir\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123928 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123955 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-modprobe-d\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123974 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-log-socket\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.123995 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9f7d1536-9b78-4f13-ae4e-349db78d213e-host-slash\") pod \"iptables-alerter-s2zcf\" (UID: \"9f7d1536-9b78-4f13-ae4e-349db78d213e\") " pod="openshift-network-operator/iptables-alerter-s2zcf" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124010 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-cni-netd\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124025 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124041 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-config\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124061 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124057 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ab83526-361f-41de-b480-79fc45d7373a-metrics-client-ca\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124066 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-modprobe-d\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124076 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-config-out\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124065 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9f7d1536-9b78-4f13-ae4e-349db78d213e-host-slash\") pod \"iptables-alerter-s2zcf\" (UID: \"9f7d1536-9b78-4f13-ae4e-349db78d213e\") " pod="openshift-network-operator/iptables-alerter-s2zcf" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124099 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-log-socket\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124105 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124125 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-cni-netd\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124107 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-metrics-server-audit-profiles\") pod \"metrics-server-748ff79954-pr5lc\" (UID: \"bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271\") " pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124133 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-host\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124141 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-config-out\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124159 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4ab83526-361f-41de-b480-79fc45d7373a-telemeter-client-tls\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124210 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-host\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124238 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124341 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4ab83526-361f-41de-b480-79fc45d7373a-federate-client-tls\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124372 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124389 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124407 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/16659cac-4c32-4a94-90aa-8be0c6be33b2-root\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124422 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124344 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124345 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4ab83526-361f-41de-b480-79fc45d7373a-telemeter-client-tls\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124435 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-host-var-lib-cni-multus\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124461 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-config\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124478 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4ab83526-361f-41de-b480-79fc45d7373a-federate-client-tls\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124478 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-client-ca-bundle\") pod \"metrics-server-748ff79954-pr5lc\" (UID: \"bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271\") " pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124534 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbcgk\" (UniqueName: \"kubernetes.io/projected/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-kube-api-access-tbcgk\") pod \"metrics-server-748ff79954-pr5lc\" (UID: \"bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271\") " pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124549 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16659cac-4c32-4a94-90aa-8be0c6be33b2-metrics-client-ca\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124562 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-host-run-netns\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124577 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-multus-conf-dir\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124592 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-run-ovn-kubernetes\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124591 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-host-var-lib-cni-multus\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124608 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124605 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124649 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/57272665-8b2d-4854-bcb6-98f662716f4c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-ztrlc\" (UID: \"57272665-8b2d-4854-bcb6-98f662716f4c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124661 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124669 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-host-run-netns\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124688 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/57272665-8b2d-4854-bcb6-98f662716f4c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-ztrlc\" (UID: \"57272665-8b2d-4854-bcb6-98f662716f4c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124717 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124719 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e5e3933f-6610-46a7-934a-8ce205da98d9-device-dir\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124759 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-lib-modules\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124788 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d6ebb345-a666-4ece-9afe-c1be0a44867b-secret-thanos-querier-tls\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124808 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/57272665-8b2d-4854-bcb6-98f662716f4c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-ztrlc\" (UID: \"57272665-8b2d-4854-bcb6-98f662716f4c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124719 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-client-ca-bundle\") pod \"metrics-server-748ff79954-pr5lc\" (UID: \"bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271\") " pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124847 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-run-ovn-kubernetes\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124817 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cd03c8ce-341a-4afe-93a1-2280392dd4f8-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-kf7dm\" (UID: \"cd03c8ce-341a-4afe-93a1-2280392dd4f8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-kf7dm" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124870 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-multus-conf-dir\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124883 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f6b477e-a0fd-49c8-896d-c268a6c06d3f-host\") pod \"node-ca-wj6dr\" (UID: \"1f6b477e-a0fd-49c8-896d-c268a6c06d3f\") " pod="openshift-image-registry/node-ca-wj6dr" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124913 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-sysconfig\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.124941 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-config-volume\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.125003 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d6ebb345-a666-4ece-9afe-c1be0a44867b-secret-thanos-querier-tls\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.125080 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-lib-modules\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.125113 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f6b477e-a0fd-49c8-896d-c268a6c06d3f-host\") pod \"node-ca-wj6dr\" (UID: \"1f6b477e-a0fd-49c8-896d-c268a6c06d3f\") " pod="openshift-image-registry/node-ca-wj6dr" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.125148 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-sysconfig\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.125175 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cd03c8ce-341a-4afe-93a1-2280392dd4f8-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-kf7dm\" (UID: \"cd03c8ce-341a-4afe-93a1-2280392dd4f8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-kf7dm" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.125248 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e5e3933f-6610-46a7-934a-8ce205da98d9-device-dir\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.125335 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/57272665-8b2d-4854-bcb6-98f662716f4c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-ztrlc\" (UID: \"57272665-8b2d-4854-bcb6-98f662716f4c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.125423 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-config-volume\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.125487 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16659cac-4c32-4a94-90aa-8be0c6be33b2-metrics-client-ca\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.125525 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.125534 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/57272665-8b2d-4854-bcb6-98f662716f4c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-ztrlc\" (UID: \"57272665-8b2d-4854-bcb6-98f662716f4c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.125553 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6ebb345-a666-4ece-9afe-c1be0a44867b-tmp\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.125565 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/57272665-8b2d-4854-bcb6-98f662716f4c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-ztrlc\" (UID: \"57272665-8b2d-4854-bcb6-98f662716f4c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.125568 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-multus-cni-dir\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.125594 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-cnibin\") pod \"multus-additional-cni-plugins-q5p48\" (UID: \"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab\") " pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.125637 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-multus-cni-dir\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.125640 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.125659 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab-cnibin\") pod \"multus-additional-cni-plugins-q5p48\" (UID: \"96ce4e9b-b61b-4769-bbc9-7e0b4174d2ab\") " pod="openshift-multus/multus-additional-cni-plugins-q5p48" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.125671 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a040e5d8-7316-4a37-a045-9eb7c2eaf607-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-jf99x\" (UID: \"a040e5d8-7316-4a37-a045-9eb7c2eaf607\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jf99x" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.125678 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6ebb345-a666-4ece-9afe-c1be0a44867b-tmp\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.125700 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a040e5d8-7316-4a37-a045-9eb7c2eaf607-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-jf99x\" (UID: \"a040e5d8-7316-4a37-a045-9eb7c2eaf607\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jf99x" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.125728 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16659cac-4c32-4a94-90aa-8be0c6be33b2-sys\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.125754 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94vqb\" (UniqueName: \"kubernetes.io/projected/4ab83526-361f-41de-b480-79fc45d7373a-kube-api-access-94vqb\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.125778 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.125780 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.125830 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d6ebb345-a666-4ece-9afe-c1be0a44867b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.125874 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a040e5d8-7316-4a37-a045-9eb7c2eaf607-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-jf99x\" (UID: \"a040e5d8-7316-4a37-a045-9eb7c2eaf607\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jf99x" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.125934 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.125965 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-host-var-lib-kubelet\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.125991 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e5e3933f-6610-46a7-934a-8ce205da98d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.126026 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-host-var-lib-kubelet\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.126042 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d6ebb345-a666-4ece-9afe-c1be0a44867b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.126044 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e5e3933f-6610-46a7-934a-8ce205da98d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-jn7gb\" (UID: \"e5e3933f-6610-46a7-934a-8ce205da98d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn7gb" Nov 04 03:17:56.126492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.126068 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a040e5d8-7316-4a37-a045-9eb7c2eaf607-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-jf99x\" (UID: \"a040e5d8-7316-4a37-a045-9eb7c2eaf607\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jf99x" Nov 04 03:17:56.131302 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.126698 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4ab83526-361f-41de-b480-79fc45d7373a-secret-telemeter-client\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:56.131302 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.126487 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4ab83526-361f-41de-b480-79fc45d7373a-secret-telemeter-client\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:56.131302 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.127077 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ab83526-361f-41de-b480-79fc45d7373a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:56.131302 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.127165 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rmmf\" (UniqueName: \"kubernetes.io/projected/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-kube-api-access-9rmmf\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.131302 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.127208 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcpzg\" (UniqueName: \"kubernetes.io/projected/16659cac-4c32-4a94-90aa-8be0c6be33b2-kube-api-access-fcpzg\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:56.131302 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.127243 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-sysctl-d\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:17:56.131302 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.127273 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/16659cac-4c32-4a94-90aa-8be0c6be33b2-node-exporter-wtmp\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:56.131302 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.127318 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/16659cac-4c32-4a94-90aa-8be0c6be33b2-node-exporter-accelerators-collector-config\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:56.131302 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.127352 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/834b5587-1820-4089-a925-ab992fe654b4-kubelet-config\") pod \"global-pull-secret-syncer-8rmjv\" (UID: \"834b5587-1820-4089-a925-ab992fe654b4\") " pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:17:56.131302 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.127386 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-slash\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.131302 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.127426 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:56.131302 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.127455 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f99pm\" (UniqueName: \"kubernetes.io/projected/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-kube-api-access-f99pm\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:56.131302 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.127489 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/57272665-8b2d-4854-bcb6-98f662716f4c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-ztrlc\" (UID: \"57272665-8b2d-4854-bcb6-98f662716f4c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc" Nov 04 03:17:56.131302 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.127524 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7c4r\" (UniqueName: \"kubernetes.io/projected/57272665-8b2d-4854-bcb6-98f662716f4c-kube-api-access-s7c4r\") pod \"kube-state-metrics-5dc8d7d89b-ztrlc\" (UID: \"57272665-8b2d-4854-bcb6-98f662716f4c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc" Nov 04 03:17:56.131302 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.127555 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-host-var-lib-cni-bin\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.131302 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.127581 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-var-lib-kubelet\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:17:56.131302 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.127612 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-systemd-units\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.131302 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.127741 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-systemd-units\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.131302 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.128046 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ab83526-361f-41de-b480-79fc45d7373a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:56.131302 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.128686 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-etc-sysctl-d\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:17:56.131302 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.128942 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/16659cac-4c32-4a94-90aa-8be0c6be33b2-node-exporter-accelerators-collector-config\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:56.131302 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.129026 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/834b5587-1820-4089-a925-ab992fe654b4-kubelet-config\") pod \"global-pull-secret-syncer-8rmjv\" (UID: \"834b5587-1820-4089-a925-ab992fe654b4\") " pod="kube-system/global-pull-secret-syncer-8rmjv" Nov 04 03:17:56.131302 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.129079 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/326a1269-c427-4470-86a6-70ee348c8791-host-slash\") pod \"ovnkube-node-bqm4b\" (UID: \"326a1269-c427-4470-86a6-70ee348c8791\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:17:56.131302 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.129306 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:56.131302 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.129795 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/57272665-8b2d-4854-bcb6-98f662716f4c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-ztrlc\" (UID: \"57272665-8b2d-4854-bcb6-98f662716f4c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc" Nov 04 03:17:56.131302 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.130083 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca254b76-1d53-44aa-81e5-e40a81265145-host-var-lib-cni-bin\") pod \"multus-t2twg\" (UID: \"ca254b76-1d53-44aa-81e5-e40a81265145\") " pod="openshift-multus/multus-t2twg" Nov 04 03:17:56.131302 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.130161 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca5887c7-01d6-44ee-ac30-8b4374663d3e-var-lib-kubelet\") pod \"tuned-7qqx5\" (UID: \"ca5887c7-01d6-44ee-ac30-8b4374663d3e\") " pod="openshift-cluster-node-tuning-operator/tuned-7qqx5" Nov 04 03:17:56.188947 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.188916 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mr7j\" (UniqueName: \"kubernetes.io/projected/d6ebb345-a666-4ece-9afe-c1be0a44867b-kube-api-access-5mr7j\") pod \"thanos-querier-7c4874f885-hqvmr\" (UID: \"d6ebb345-a666-4ece-9afe-c1be0a44867b\") " pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:56.191349 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.191328 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvm8t\" (UniqueName: \"kubernetes.io/projected/a040e5d8-7316-4a37-a045-9eb7c2eaf607-kube-api-access-pvm8t\") pod \"openshift-state-metrics-7bd7489957-jf99x\" (UID: \"a040e5d8-7316-4a37-a045-9eb7c2eaf607\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jf99x" Nov 04 03:17:56.191667 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.191646 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtnzk\" (UniqueName: \"kubernetes.io/projected/cd03c8ce-341a-4afe-93a1-2280392dd4f8-kube-api-access-gtnzk\") pod \"monitoring-plugin-77f4f76594-kf7dm\" (UID: \"cd03c8ce-341a-4afe-93a1-2280392dd4f8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-kf7dm" Nov 04 03:17:56.191781 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.191764 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rmmf\" (UniqueName: \"kubernetes.io/projected/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-kube-api-access-9rmmf\") pod \"prometheus-k8s-0\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:56.192482 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.192461 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbcgk\" (UniqueName: \"kubernetes.io/projected/bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271-kube-api-access-tbcgk\") pod \"metrics-server-748ff79954-pr5lc\" (UID: \"bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271\") " pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" Nov 04 03:17:56.194119 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.194103 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94vqb\" (UniqueName: \"kubernetes.io/projected/4ab83526-361f-41de-b480-79fc45d7373a-kube-api-access-94vqb\") pod \"telemeter-client-745f9d6695-rlnsq\" (UID: \"4ab83526-361f-41de-b480-79fc45d7373a\") " pod="openshift-monitoring/telemeter-client-745f9d6695-rlnsq" Nov 04 03:17:56.195750 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.195730 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcpzg\" (UniqueName: \"kubernetes.io/projected/16659cac-4c32-4a94-90aa-8be0c6be33b2-kube-api-access-fcpzg\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:56.195998 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.195983 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f99pm\" (UniqueName: \"kubernetes.io/projected/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-kube-api-access-f99pm\") pod \"alertmanager-main-0\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:17:56.196553 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.196538 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7c4r\" (UniqueName: \"kubernetes.io/projected/57272665-8b2d-4854-bcb6-98f662716f4c-kube-api-access-s7c4r\") pod \"kube-state-metrics-5dc8d7d89b-ztrlc\" (UID: \"57272665-8b2d-4854-bcb6-98f662716f4c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ztrlc" Nov 04 03:17:56.228476 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.228445 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/16659cac-4c32-4a94-90aa-8be0c6be33b2-root\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:56.228663 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.228486 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16659cac-4c32-4a94-90aa-8be0c6be33b2-sys\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:56.228663 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.228506 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/16659cac-4c32-4a94-90aa-8be0c6be33b2-node-exporter-wtmp\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:56.228663 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.228559 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/16659cac-4c32-4a94-90aa-8be0c6be33b2-root\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:56.228663 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.228564 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16659cac-4c32-4a94-90aa-8be0c6be33b2-sys\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:56.228663 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:56.228595 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/16659cac-4c32-4a94-90aa-8be0c6be33b2-node-exporter-wtmp\") pod \"node-exporter-7c9tl\" (UID: \"16659cac-4c32-4a94-90aa-8be0c6be33b2\") " pod="openshift-monitoring/node-exporter-7c9tl" Nov 04 03:17:57.118665 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:57.118614 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7c4874f885-hqvmr" Nov 04 03:17:58.390542 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:58.390511 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:58.409879 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:58.409844 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:17:58.936556 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:58.936523 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jg2pw" Nov 04 03:17:59.050447 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:17:59.050415 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pmv6n" Nov 04 03:18:00.076797 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:00.076759 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:18:00.815166 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:00.815135 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:18:02.047094 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:02.047063 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:18:02.047559 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:02.047279 5953 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 03:18:03.208231 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:03.208199 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kf7dm" Nov 04 03:18:09.036170 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:09.036085 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:11.249688 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:11.249655 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-748ff79954-pr5lc" Nov 04 03:18:16.383893 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:18:16.383853 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:18:18.494062 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:18.494031 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bqm4b" Nov 04 03:18:19.944462 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:19.944432 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 04 03:18:19.944928 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:19.944811 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" containerName="kube-rbac-proxy-metric" containerID="cri-o://cb5b83e3d9d660b2441013aee8be5c36b6faed092a60491930b9de1a974dd478" gracePeriod=120 Nov 04 03:18:19.944928 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:19.944852 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" containerName="prom-label-proxy" containerID="cri-o://c9a65ccf157d7695f25dd196151bee5ecfbb9b0ad205c85e7060d0537fd40706" gracePeriod=120 Nov 04 03:18:19.944928 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:19.944877 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" containerName="kube-rbac-proxy-web" containerID="cri-o://b1d85abe163b8f70f8552f5dc26ba32f89375f6af2c6043ae5f5ef1c71573718" gracePeriod=120 Nov 04 03:18:19.945119 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:19.944900 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" containerName="config-reloader" containerID="cri-o://a3be12ddcd9acb0895e6f0a5b93c51bf1431a92793e1cbc155c89fbb163ff0b0" gracePeriod=120 Nov 04 03:18:19.945119 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:19.944810 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" containerName="alertmanager" containerID="cri-o://8254ce7d4c3adaa19ae449cdd41cf2b74f62812a3853535de6ade8f79389fa13" gracePeriod=120 Nov 04 03:18:19.945119 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:19.945074 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" containerName="kube-rbac-proxy" containerID="cri-o://39c4768b174ad259edc7e17463fadf1eee93694e930100e24c578bd0f207db44" gracePeriod=120 Nov 04 03:18:21.025397 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.025372 5953 generic.go:356] "Generic (PLEG): container finished" podID="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" containerID="c9a65ccf157d7695f25dd196151bee5ecfbb9b0ad205c85e7060d0537fd40706" exitCode=0 Nov 04 03:18:21.025397 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.025394 5953 generic.go:356] "Generic (PLEG): container finished" podID="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" containerID="cb5b83e3d9d660b2441013aee8be5c36b6faed092a60491930b9de1a974dd478" exitCode=0 Nov 04 03:18:21.025783 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.025402 5953 generic.go:356] "Generic (PLEG): container finished" podID="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" containerID="39c4768b174ad259edc7e17463fadf1eee93694e930100e24c578bd0f207db44" exitCode=0 Nov 04 03:18:21.025783 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.025411 5953 generic.go:356] "Generic (PLEG): container finished" podID="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" containerID="a3be12ddcd9acb0895e6f0a5b93c51bf1431a92793e1cbc155c89fbb163ff0b0" exitCode=0 Nov 04 03:18:21.025783 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.025418 5953 generic.go:356] "Generic (PLEG): container finished" podID="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" containerID="8254ce7d4c3adaa19ae449cdd41cf2b74f62812a3853535de6ade8f79389fa13" exitCode=0 Nov 04 03:18:21.025783 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.025441 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31","Type":"ContainerDied","Data":"c9a65ccf157d7695f25dd196151bee5ecfbb9b0ad205c85e7060d0537fd40706"} Nov 04 03:18:21.025783 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.025499 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31","Type":"ContainerDied","Data":"cb5b83e3d9d660b2441013aee8be5c36b6faed092a60491930b9de1a974dd478"} Nov 04 03:18:21.025783 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.025527 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31","Type":"ContainerDied","Data":"39c4768b174ad259edc7e17463fadf1eee93694e930100e24c578bd0f207db44"} Nov 04 03:18:21.025783 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.025552 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31","Type":"ContainerDied","Data":"a3be12ddcd9acb0895e6f0a5b93c51bf1431a92793e1cbc155c89fbb163ff0b0"} Nov 04 03:18:21.025783 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.025575 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31","Type":"ContainerDied","Data":"8254ce7d4c3adaa19ae449cdd41cf2b74f62812a3853535de6ade8f79389fa13"} Nov 04 03:18:21.200067 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.200044 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:21.359387 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.359302 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-tls-assets\" (UniqueName: \"kubernetes.io/projected/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-tls-assets\") pod \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " Nov 04 03:18:21.359387 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.359337 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-alertmanager-main-db\") pod \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " Nov 04 03:18:21.359387 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.359366 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-kube-api-access-f99pm\" (UniqueName: \"kubernetes.io/projected/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-kube-api-access-f99pm\") pod \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " Nov 04 03:18:21.359683 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.359393 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-config-volume\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-config-volume\") pod \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " Nov 04 03:18:21.359683 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.359439 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-cluster-tls-config\") pod \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " Nov 04 03:18:21.359683 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.359479 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-metrics-client-ca\") pod \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " Nov 04 03:18:21.359683 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.359510 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-config-out\" (UniqueName: \"kubernetes.io/empty-dir/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-config-out\") pod \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " Nov 04 03:18:21.359683 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.359549 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-main-tls\") pod \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " Nov 04 03:18:21.359683 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.359588 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-web-config\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-web-config\") pod \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " Nov 04 03:18:21.359683 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.359652 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-kube-rbac-proxy-metric\") pod \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " Nov 04 03:18:21.360043 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.359729 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-alertmanager-trusted-ca-bundle\") pod \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " Nov 04 03:18:21.360043 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.359762 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-alertmanager-main-db" pod "d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" (UID: "d10b8919-8b30-4ee5-b4f9-2ea7794bfd31"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:18:21.360043 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.359786 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-kube-rbac-proxy-web\") pod \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " Nov 04 03:18:21.360043 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.359816 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-kube-rbac-proxy\") pod \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\" (UID: \"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31\") " Nov 04 03:18:21.360461 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.360061 5953 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-alertmanager-main-db\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:21.360461 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.360136 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-alertmanager-trusted-ca-bundle" pod "d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" (UID: "d10b8919-8b30-4ee5-b4f9-2ea7794bfd31"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:18:21.360574 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.360531 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-metrics-client-ca" pod "d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" (UID: "d10b8919-8b30-4ee5-b4f9-2ea7794bfd31"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:18:21.363608 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.363551 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-config-volume" pod "d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" (UID: "d10b8919-8b30-4ee5-b4f9-2ea7794bfd31"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:18:21.363798 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.363643 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-tls-assets" pod "d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" (UID: "d10b8919-8b30-4ee5-b4f9-2ea7794bfd31"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:18:21.363798 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.363709 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-kube-rbac-proxy-metric" pod "d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" (UID: "d10b8919-8b30-4ee5-b4f9-2ea7794bfd31"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:18:21.363965 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.363797 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-kube-rbac-proxy" pod "d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" (UID: "d10b8919-8b30-4ee5-b4f9-2ea7794bfd31"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:18:21.363965 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.363937 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-kube-api-access-f99pm" pod "d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" (UID: "d10b8919-8b30-4ee5-b4f9-2ea7794bfd31"). InnerVolumeSpecName "kube-api-access-f99pm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:18:21.364078 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.363978 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-kube-rbac-proxy-web" pod "d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" (UID: "d10b8919-8b30-4ee5-b4f9-2ea7794bfd31"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:18:21.364740 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.364707 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-main-tls" pod "d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" (UID: "d10b8919-8b30-4ee5-b4f9-2ea7794bfd31"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:18:21.364841 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.364802 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-config-out" pod "d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" (UID: "d10b8919-8b30-4ee5-b4f9-2ea7794bfd31"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:18:21.367920 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.367814 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-cluster-tls-config" pod "d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" (UID: "d10b8919-8b30-4ee5-b4f9-2ea7794bfd31"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:18:21.374361 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.374338 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-web-config" pod "d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" (UID: "d10b8919-8b30-4ee5-b4f9-2ea7794bfd31"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:18:21.461006 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.460970 5953 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-web-config\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:21.461006 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.461000 5953 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:21.461006 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.461014 5953 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:21.461278 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.461023 5953 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:21.461278 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.461032 5953 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:21.461278 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.461041 5953 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-tls-assets\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:21.461278 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.461050 5953 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f99pm\" (UniqueName: \"kubernetes.io/projected/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-kube-api-access-f99pm\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:21.461278 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.461058 5953 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-config-volume\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:21.461278 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.461066 5953 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-cluster-tls-config\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:21.461278 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.461074 5953 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-metrics-client-ca\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:21.461278 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.461082 5953 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-config-out\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:21.461278 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:21.461090 5953 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31-secret-alertmanager-main-tls\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:22.032096 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.032066 5953 generic.go:356] "Generic (PLEG): container finished" podID="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" containerID="b1d85abe163b8f70f8552f5dc26ba32f89375f6af2c6043ae5f5ef1c71573718" exitCode=0 Nov 04 03:18:22.032460 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.032130 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31","Type":"ContainerDied","Data":"b1d85abe163b8f70f8552f5dc26ba32f89375f6af2c6043ae5f5ef1c71573718"} Nov 04 03:18:22.032460 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.032165 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.032460 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.032186 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d10b8919-8b30-4ee5-b4f9-2ea7794bfd31","Type":"ContainerDied","Data":"6132bb22558201eafc6fe42434f09c31763011a7226766084223d4783a699f8f"} Nov 04 03:18:22.032460 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.032216 5953 scope.go:122] "RemoveContainer" containerID="c9a65ccf157d7695f25dd196151bee5ecfbb9b0ad205c85e7060d0537fd40706" Nov 04 03:18:22.040829 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.040808 5953 scope.go:122] "RemoveContainer" containerID="cb5b83e3d9d660b2441013aee8be5c36b6faed092a60491930b9de1a974dd478" Nov 04 03:18:22.049244 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.049224 5953 scope.go:122] "RemoveContainer" containerID="39c4768b174ad259edc7e17463fadf1eee93694e930100e24c578bd0f207db44" Nov 04 03:18:22.056389 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.056365 5953 scope.go:122] "RemoveContainer" containerID="b1d85abe163b8f70f8552f5dc26ba32f89375f6af2c6043ae5f5ef1c71573718" Nov 04 03:18:22.065081 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.064901 5953 scope.go:122] "RemoveContainer" containerID="a3be12ddcd9acb0895e6f0a5b93c51bf1431a92793e1cbc155c89fbb163ff0b0" Nov 04 03:18:22.072044 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.072026 5953 scope.go:122] "RemoveContainer" containerID="8254ce7d4c3adaa19ae449cdd41cf2b74f62812a3853535de6ade8f79389fa13" Nov 04 03:18:22.078728 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.078707 5953 scope.go:122] "RemoveContainer" containerID="f4ac97bd1d0e01b9f860c3ffeb1a6073e2aa15efea50981c2053c645efdbe607" Nov 04 03:18:22.085968 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.085951 5953 scope.go:122] "RemoveContainer" containerID="c9a65ccf157d7695f25dd196151bee5ecfbb9b0ad205c85e7060d0537fd40706" Nov 04 03:18:22.086239 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:18:22.086221 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9a65ccf157d7695f25dd196151bee5ecfbb9b0ad205c85e7060d0537fd40706\": container with ID starting with c9a65ccf157d7695f25dd196151bee5ecfbb9b0ad205c85e7060d0537fd40706 not found: ID does not exist" containerID="c9a65ccf157d7695f25dd196151bee5ecfbb9b0ad205c85e7060d0537fd40706" Nov 04 03:18:22.086290 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.086248 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9a65ccf157d7695f25dd196151bee5ecfbb9b0ad205c85e7060d0537fd40706"} err="failed to get container status \"c9a65ccf157d7695f25dd196151bee5ecfbb9b0ad205c85e7060d0537fd40706\": rpc error: code = NotFound desc = could not find container \"c9a65ccf157d7695f25dd196151bee5ecfbb9b0ad205c85e7060d0537fd40706\": container with ID starting with c9a65ccf157d7695f25dd196151bee5ecfbb9b0ad205c85e7060d0537fd40706 not found: ID does not exist" Nov 04 03:18:22.086290 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.086279 5953 scope.go:122] "RemoveContainer" containerID="cb5b83e3d9d660b2441013aee8be5c36b6faed092a60491930b9de1a974dd478" Nov 04 03:18:22.086489 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:18:22.086468 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb5b83e3d9d660b2441013aee8be5c36b6faed092a60491930b9de1a974dd478\": container with ID starting with cb5b83e3d9d660b2441013aee8be5c36b6faed092a60491930b9de1a974dd478 not found: ID does not exist" containerID="cb5b83e3d9d660b2441013aee8be5c36b6faed092a60491930b9de1a974dd478" Nov 04 03:18:22.086553 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.086499 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb5b83e3d9d660b2441013aee8be5c36b6faed092a60491930b9de1a974dd478"} err="failed to get container status \"cb5b83e3d9d660b2441013aee8be5c36b6faed092a60491930b9de1a974dd478\": rpc error: code = NotFound desc = could not find container \"cb5b83e3d9d660b2441013aee8be5c36b6faed092a60491930b9de1a974dd478\": container with ID starting with cb5b83e3d9d660b2441013aee8be5c36b6faed092a60491930b9de1a974dd478 not found: ID does not exist" Nov 04 03:18:22.086553 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.086521 5953 scope.go:122] "RemoveContainer" containerID="39c4768b174ad259edc7e17463fadf1eee93694e930100e24c578bd0f207db44" Nov 04 03:18:22.086803 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:18:22.086787 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39c4768b174ad259edc7e17463fadf1eee93694e930100e24c578bd0f207db44\": container with ID starting with 39c4768b174ad259edc7e17463fadf1eee93694e930100e24c578bd0f207db44 not found: ID does not exist" containerID="39c4768b174ad259edc7e17463fadf1eee93694e930100e24c578bd0f207db44" Nov 04 03:18:22.086852 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.086810 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39c4768b174ad259edc7e17463fadf1eee93694e930100e24c578bd0f207db44"} err="failed to get container status \"39c4768b174ad259edc7e17463fadf1eee93694e930100e24c578bd0f207db44\": rpc error: code = NotFound desc = could not find container \"39c4768b174ad259edc7e17463fadf1eee93694e930100e24c578bd0f207db44\": container with ID starting with 39c4768b174ad259edc7e17463fadf1eee93694e930100e24c578bd0f207db44 not found: ID does not exist" Nov 04 03:18:22.086852 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.086826 5953 scope.go:122] "RemoveContainer" containerID="b1d85abe163b8f70f8552f5dc26ba32f89375f6af2c6043ae5f5ef1c71573718" Nov 04 03:18:22.087041 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:18:22.087022 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1d85abe163b8f70f8552f5dc26ba32f89375f6af2c6043ae5f5ef1c71573718\": container with ID starting with b1d85abe163b8f70f8552f5dc26ba32f89375f6af2c6043ae5f5ef1c71573718 not found: ID does not exist" containerID="b1d85abe163b8f70f8552f5dc26ba32f89375f6af2c6043ae5f5ef1c71573718" Nov 04 03:18:22.087083 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.087048 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1d85abe163b8f70f8552f5dc26ba32f89375f6af2c6043ae5f5ef1c71573718"} err="failed to get container status \"b1d85abe163b8f70f8552f5dc26ba32f89375f6af2c6043ae5f5ef1c71573718\": rpc error: code = NotFound desc = could not find container \"b1d85abe163b8f70f8552f5dc26ba32f89375f6af2c6043ae5f5ef1c71573718\": container with ID starting with b1d85abe163b8f70f8552f5dc26ba32f89375f6af2c6043ae5f5ef1c71573718 not found: ID does not exist" Nov 04 03:18:22.087083 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.087064 5953 scope.go:122] "RemoveContainer" containerID="a3be12ddcd9acb0895e6f0a5b93c51bf1431a92793e1cbc155c89fbb163ff0b0" Nov 04 03:18:22.087284 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:18:22.087268 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3be12ddcd9acb0895e6f0a5b93c51bf1431a92793e1cbc155c89fbb163ff0b0\": container with ID starting with a3be12ddcd9acb0895e6f0a5b93c51bf1431a92793e1cbc155c89fbb163ff0b0 not found: ID does not exist" containerID="a3be12ddcd9acb0895e6f0a5b93c51bf1431a92793e1cbc155c89fbb163ff0b0" Nov 04 03:18:22.087353 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.087292 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3be12ddcd9acb0895e6f0a5b93c51bf1431a92793e1cbc155c89fbb163ff0b0"} err="failed to get container status \"a3be12ddcd9acb0895e6f0a5b93c51bf1431a92793e1cbc155c89fbb163ff0b0\": rpc error: code = NotFound desc = could not find container \"a3be12ddcd9acb0895e6f0a5b93c51bf1431a92793e1cbc155c89fbb163ff0b0\": container with ID starting with a3be12ddcd9acb0895e6f0a5b93c51bf1431a92793e1cbc155c89fbb163ff0b0 not found: ID does not exist" Nov 04 03:18:22.087353 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.087310 5953 scope.go:122] "RemoveContainer" containerID="8254ce7d4c3adaa19ae449cdd41cf2b74f62812a3853535de6ade8f79389fa13" Nov 04 03:18:22.087546 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:18:22.087530 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8254ce7d4c3adaa19ae449cdd41cf2b74f62812a3853535de6ade8f79389fa13\": container with ID starting with 8254ce7d4c3adaa19ae449cdd41cf2b74f62812a3853535de6ade8f79389fa13 not found: ID does not exist" containerID="8254ce7d4c3adaa19ae449cdd41cf2b74f62812a3853535de6ade8f79389fa13" Nov 04 03:18:22.087585 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.087550 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8254ce7d4c3adaa19ae449cdd41cf2b74f62812a3853535de6ade8f79389fa13"} err="failed to get container status \"8254ce7d4c3adaa19ae449cdd41cf2b74f62812a3853535de6ade8f79389fa13\": rpc error: code = NotFound desc = could not find container \"8254ce7d4c3adaa19ae449cdd41cf2b74f62812a3853535de6ade8f79389fa13\": container with ID starting with 8254ce7d4c3adaa19ae449cdd41cf2b74f62812a3853535de6ade8f79389fa13 not found: ID does not exist" Nov 04 03:18:22.087585 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.087563 5953 scope.go:122] "RemoveContainer" containerID="f4ac97bd1d0e01b9f860c3ffeb1a6073e2aa15efea50981c2053c645efdbe607" Nov 04 03:18:22.087808 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:18:22.087790 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4ac97bd1d0e01b9f860c3ffeb1a6073e2aa15efea50981c2053c645efdbe607\": container with ID starting with f4ac97bd1d0e01b9f860c3ffeb1a6073e2aa15efea50981c2053c645efdbe607 not found: ID does not exist" containerID="f4ac97bd1d0e01b9f860c3ffeb1a6073e2aa15efea50981c2053c645efdbe607" Nov 04 03:18:22.087858 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.087815 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4ac97bd1d0e01b9f860c3ffeb1a6073e2aa15efea50981c2053c645efdbe607"} err="failed to get container status \"f4ac97bd1d0e01b9f860c3ffeb1a6073e2aa15efea50981c2053c645efdbe607\": rpc error: code = NotFound desc = could not find container \"f4ac97bd1d0e01b9f860c3ffeb1a6073e2aa15efea50981c2053c645efdbe607\": container with ID starting with f4ac97bd1d0e01b9f860c3ffeb1a6073e2aa15efea50981c2053c645efdbe607 not found: ID does not exist" Nov 04 03:18:22.250154 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.250121 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 04 03:18:22.300866 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.300780 5953 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 04 03:18:22.301169 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.301150 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 04 03:18:22.301441 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.301430 5953 cpu_manager.go:414] "containerMap: removing container" podUID="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" containerName="kube-rbac-proxy-metric" Nov 04 03:18:22.301475 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.301443 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" containerName="kube-rbac-proxy-metric" Nov 04 03:18:22.301475 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.301461 5953 cpu_manager.go:414] "containerMap: removing container" podUID="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" containerName="prom-label-proxy" Nov 04 03:18:22.301475 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.301466 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" containerName="prom-label-proxy" Nov 04 03:18:22.301475 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.301473 5953 cpu_manager.go:414] "containerMap: removing container" podUID="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" containerName="kube-rbac-proxy" Nov 04 03:18:22.301586 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.301477 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" containerName="kube-rbac-proxy" Nov 04 03:18:22.301586 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.301485 5953 cpu_manager.go:414] "containerMap: removing container" podUID="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" containerName="alertmanager" Nov 04 03:18:22.301586 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.301490 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" containerName="alertmanager" Nov 04 03:18:22.301586 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.301499 5953 cpu_manager.go:414] "containerMap: removing container" podUID="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" containerName="init-config-reloader" Nov 04 03:18:22.301586 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.301504 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" containerName="init-config-reloader" Nov 04 03:18:22.301586 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.301518 5953 cpu_manager.go:414] "containerMap: removing container" podUID="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" containerName="kube-rbac-proxy-web" Nov 04 03:18:22.301586 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.301523 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" containerName="kube-rbac-proxy-web" Nov 04 03:18:22.301586 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.301529 5953 cpu_manager.go:414] "containerMap: removing container" podUID="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" containerName="config-reloader" Nov 04 03:18:22.301586 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.301533 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" containerName="config-reloader" Nov 04 03:18:22.301586 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.301580 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" containerName="kube-rbac-proxy-metric" Nov 04 03:18:22.301586 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.301588 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" containerName="prom-label-proxy" Nov 04 03:18:22.301911 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.301593 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" containerName="kube-rbac-proxy" Nov 04 03:18:22.301911 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.301599 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" containerName="alertmanager" Nov 04 03:18:22.301911 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.301605 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" containerName="init-config-reloader" Nov 04 03:18:22.301911 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.301610 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" containerName="kube-rbac-proxy-web" Nov 04 03:18:22.301911 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.301630 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" containerName="config-reloader" Nov 04 03:18:22.303685 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.303671 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.356836 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.356798 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 04 03:18:22.357062 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.356995 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 04 03:18:22.357062 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.357017 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 04 03:18:22.357062 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.357020 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 04 03:18:22.357062 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.357026 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 04 03:18:22.357270 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.357074 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 04 03:18:22.357270 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.357089 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 04 03:18:22.357270 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.357080 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-mwn26\"" Nov 04 03:18:22.357270 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.357174 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 04 03:18:22.407221 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.407194 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 04 03:18:22.469010 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.468977 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnslm\" (UniqueName: \"kubernetes.io/projected/c124d431-5af1-4978-9a20-cd4637befe01-kube-api-access-tnslm\") pod \"alertmanager-main-0\" (UID: \"c124d431-5af1-4978-9a20-cd4637befe01\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.469160 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.469019 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c124d431-5af1-4978-9a20-cd4637befe01-config-out\") pod \"alertmanager-main-0\" (UID: \"c124d431-5af1-4978-9a20-cd4637befe01\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.469160 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.469035 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c124d431-5af1-4978-9a20-cd4637befe01-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c124d431-5af1-4978-9a20-cd4637befe01\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.469160 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.469120 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c124d431-5af1-4978-9a20-cd4637befe01-web-config\") pod \"alertmanager-main-0\" (UID: \"c124d431-5af1-4978-9a20-cd4637befe01\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.469160 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.469152 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c124d431-5af1-4978-9a20-cd4637befe01-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c124d431-5af1-4978-9a20-cd4637befe01\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.469369 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.469175 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c124d431-5af1-4978-9a20-cd4637befe01-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c124d431-5af1-4978-9a20-cd4637befe01\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.469369 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.469223 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c124d431-5af1-4978-9a20-cd4637befe01-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c124d431-5af1-4978-9a20-cd4637befe01\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.469369 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.469325 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c124d431-5af1-4978-9a20-cd4637befe01-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c124d431-5af1-4978-9a20-cd4637befe01\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.469519 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.469367 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c124d431-5af1-4978-9a20-cd4637befe01-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c124d431-5af1-4978-9a20-cd4637befe01\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.469519 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.469403 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c124d431-5af1-4978-9a20-cd4637befe01-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c124d431-5af1-4978-9a20-cd4637befe01\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.469519 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.469452 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c124d431-5af1-4978-9a20-cd4637befe01-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c124d431-5af1-4978-9a20-cd4637befe01\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.469519 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.469478 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c124d431-5af1-4978-9a20-cd4637befe01-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c124d431-5af1-4978-9a20-cd4637befe01\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.469672 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.469531 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c124d431-5af1-4978-9a20-cd4637befe01-config-volume\") pod \"alertmanager-main-0\" (UID: \"c124d431-5af1-4978-9a20-cd4637befe01\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.543798 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.543760 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 04 03:18:22.570210 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.570127 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c124d431-5af1-4978-9a20-cd4637befe01-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c124d431-5af1-4978-9a20-cd4637befe01\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.570210 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.570166 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c124d431-5af1-4978-9a20-cd4637befe01-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c124d431-5af1-4978-9a20-cd4637befe01\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.570423 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.570299 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c124d431-5af1-4978-9a20-cd4637befe01-config-volume\") pod \"alertmanager-main-0\" (UID: \"c124d431-5af1-4978-9a20-cd4637befe01\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.570423 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.570347 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnslm\" (UniqueName: \"kubernetes.io/projected/c124d431-5af1-4978-9a20-cd4637befe01-kube-api-access-tnslm\") pod \"alertmanager-main-0\" (UID: \"c124d431-5af1-4978-9a20-cd4637befe01\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.570423 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.570395 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c124d431-5af1-4978-9a20-cd4637befe01-config-out\") pod \"alertmanager-main-0\" (UID: \"c124d431-5af1-4978-9a20-cd4637befe01\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.570576 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.570424 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c124d431-5af1-4978-9a20-cd4637befe01-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c124d431-5af1-4978-9a20-cd4637befe01\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.570576 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.570454 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c124d431-5af1-4978-9a20-cd4637befe01-web-config\") pod \"alertmanager-main-0\" (UID: \"c124d431-5af1-4978-9a20-cd4637befe01\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.570576 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.570481 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c124d431-5af1-4978-9a20-cd4637befe01-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c124d431-5af1-4978-9a20-cd4637befe01\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.570576 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.570513 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c124d431-5af1-4978-9a20-cd4637befe01-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c124d431-5af1-4978-9a20-cd4637befe01\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.570576 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.570557 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c124d431-5af1-4978-9a20-cd4637befe01-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c124d431-5af1-4978-9a20-cd4637befe01\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.570841 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.570595 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c124d431-5af1-4978-9a20-cd4637befe01-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c124d431-5af1-4978-9a20-cd4637befe01\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.570841 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.570650 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c124d431-5af1-4978-9a20-cd4637befe01-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c124d431-5af1-4978-9a20-cd4637befe01\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.570841 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.570680 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c124d431-5af1-4978-9a20-cd4637befe01-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c124d431-5af1-4978-9a20-cd4637befe01\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.570841 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.570829 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c124d431-5af1-4978-9a20-cd4637befe01-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c124d431-5af1-4978-9a20-cd4637befe01\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.571047 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.570860 5953 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 04 03:18:22.571754 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.571540 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c124d431-5af1-4978-9a20-cd4637befe01-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c124d431-5af1-4978-9a20-cd4637befe01\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.571754 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.571697 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c124d431-5af1-4978-9a20-cd4637befe01-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c124d431-5af1-4978-9a20-cd4637befe01\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.574073 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.573999 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c124d431-5af1-4978-9a20-cd4637befe01-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c124d431-5af1-4978-9a20-cd4637befe01\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.574073 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.574029 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c124d431-5af1-4978-9a20-cd4637befe01-config-volume\") pod \"alertmanager-main-0\" (UID: \"c124d431-5af1-4978-9a20-cd4637befe01\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.574241 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.574106 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c124d431-5af1-4978-9a20-cd4637befe01-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c124d431-5af1-4978-9a20-cd4637befe01\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.574241 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.574198 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c124d431-5af1-4978-9a20-cd4637befe01-config-out\") pod \"alertmanager-main-0\" (UID: \"c124d431-5af1-4978-9a20-cd4637befe01\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.574343 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.574322 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c124d431-5af1-4978-9a20-cd4637befe01-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c124d431-5af1-4978-9a20-cd4637befe01\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.574597 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.574578 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c124d431-5af1-4978-9a20-cd4637befe01-web-config\") pod \"alertmanager-main-0\" (UID: \"c124d431-5af1-4978-9a20-cd4637befe01\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.574697 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.574579 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c124d431-5af1-4978-9a20-cd4637befe01-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c124d431-5af1-4978-9a20-cd4637befe01\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.575087 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.575066 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c124d431-5af1-4978-9a20-cd4637befe01-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c124d431-5af1-4978-9a20-cd4637befe01\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.575614 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.575592 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c124d431-5af1-4978-9a20-cd4637befe01-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c124d431-5af1-4978-9a20-cd4637befe01\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.631649 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.631604 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnslm\" (UniqueName: \"kubernetes.io/projected/c124d431-5af1-4978-9a20-cd4637befe01-kube-api-access-tnslm\") pod \"alertmanager-main-0\" (UID: \"c124d431-5af1-4978-9a20-cd4637befe01\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:22.853682 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.853582 5953 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d10b8919-8b30-4ee5-b4f9-2ea7794bfd31" path="/var/lib/kubelet/pods/d10b8919-8b30-4ee5-b4f9-2ea7794bfd31/volumes" Nov 04 03:18:22.918556 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:22.918521 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 04 03:18:23.396337 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:23.396306 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 04 03:18:23.399827 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:18:23.399795 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc124d431_5af1_4978_9a20_cd4637befe01.slice/crio-aa2446410229a1453cae5bde9bd8639f19160c57fce6d2c477296237bc7c00d0 WatchSource:0}: Error finding container aa2446410229a1453cae5bde9bd8639f19160c57fce6d2c477296237bc7c00d0: Status 404 returned error can't find the container with id aa2446410229a1453cae5bde9bd8639f19160c57fce6d2c477296237bc7c00d0 Nov 04 03:18:24.046410 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:24.046371 5953 generic.go:356] "Generic (PLEG): container finished" podID="c124d431-5af1-4978-9a20-cd4637befe01" containerID="bb21b02cc2acc040247b0839ef43eda23953ca6687e81a907894677799677dc3" exitCode=0 Nov 04 03:18:24.046576 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:24.046450 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c124d431-5af1-4978-9a20-cd4637befe01","Type":"ContainerDied","Data":"bb21b02cc2acc040247b0839ef43eda23953ca6687e81a907894677799677dc3"} Nov 04 03:18:24.046576 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:24.046539 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c124d431-5af1-4978-9a20-cd4637befe01","Type":"ContainerStarted","Data":"aa2446410229a1453cae5bde9bd8639f19160c57fce6d2c477296237bc7c00d0"} Nov 04 03:18:25.054092 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:25.054055 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c124d431-5af1-4978-9a20-cd4637befe01","Type":"ContainerStarted","Data":"c84162a0ee33d91a224c044ee6d437e1feb1852dca0874b57d34f18ae9256722"} Nov 04 03:18:25.054565 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:25.054117 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c124d431-5af1-4978-9a20-cd4637befe01","Type":"ContainerStarted","Data":"71c01c87875ce1225b4e42bf39dcf589f04a0a3db3370186f27a5cdb167b39e1"} Nov 04 03:18:25.054565 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:25.054142 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c124d431-5af1-4978-9a20-cd4637befe01","Type":"ContainerStarted","Data":"c1277159cc8f0e69a63fcd7b21123e27457a2da418c23464493fd5e1fba553fb"} Nov 04 03:18:25.054565 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:25.054166 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c124d431-5af1-4978-9a20-cd4637befe01","Type":"ContainerStarted","Data":"87c75c4eb6f4e52f0bc22f106e58818d3800c30db691b42f61b7475d2f042b20"} Nov 04 03:18:25.054565 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:25.054189 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c124d431-5af1-4978-9a20-cd4637befe01","Type":"ContainerStarted","Data":"36e1ff5bb31bc18d7212e8bbd3d595f2c9fdd42d5e6053e4b8d672ce77504d03"} Nov 04 03:18:25.054565 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:25.054211 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c124d431-5af1-4978-9a20-cd4637befe01","Type":"ContainerStarted","Data":"679e37bc10912f0884721375c6da58556cd911ed9f9b373aa5d9b738af4631d2"} Nov 04 03:18:25.185585 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:25.185527 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.173154826 podStartE2EDuration="3.185512063s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.357237ms" isStatefulPod=false podCreationTimestamp="2026-11-04 03:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:18:25.183455215 +0000 UTC m=+30.458085977" watchObservedRunningTime="2026-11-04 03:18:25.185512063 +0000 UTC m=+30.460142876" Nov 04 03:18:25.658917 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:25.658882 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 03:18:25.659268 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:25.659230 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" containerName="prometheus" containerID="cri-o://d5918d555e4e664d0b9dc19f3ea84d5d728df2cb00e5dad2143ea52b72766645" gracePeriod=600 Nov 04 03:18:25.659404 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:25.659235 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" containerName="kube-rbac-proxy-thanos" containerID="cri-o://4923b65f9e3377eb7546b564bc8470d59713a6761433c1ccd7fbc667e707250c" gracePeriod=600 Nov 04 03:18:25.659404 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:25.659257 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" containerName="thanos-sidecar" containerID="cri-o://8df5edc17acbce219b8ee1e04527c0a07bfa8e3ef8772a07acb7c46bbc69b457" gracePeriod=600 Nov 04 03:18:25.659404 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:25.659283 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" containerName="config-reloader" containerID="cri-o://5aea6e059cb33894f5559432ed56c5f1f503907e0955d2dfdc7419d997f5dbf8" gracePeriod=600 Nov 04 03:18:25.659404 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:25.659263 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" containerName="kube-rbac-proxy-web" containerID="cri-o://2f40b0a79469bfb66ca1723dfed6cce23aab6b2e980135a50e11335832775639" gracePeriod=600 Nov 04 03:18:25.659590 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:25.659332 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" containerName="kube-rbac-proxy" containerID="cri-o://dedb1c9816ecad494cbc4bde07bdea706f4a7aaffce47e5e101a9db1a532b062" gracePeriod=600 Nov 04 03:18:25.906975 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:25.906949 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.061583 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.061548 5953 generic.go:356] "Generic (PLEG): container finished" podID="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" containerID="4923b65f9e3377eb7546b564bc8470d59713a6761433c1ccd7fbc667e707250c" exitCode=0 Nov 04 03:18:26.061583 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.061576 5953 generic.go:356] "Generic (PLEG): container finished" podID="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" containerID="dedb1c9816ecad494cbc4bde07bdea706f4a7aaffce47e5e101a9db1a532b062" exitCode=0 Nov 04 03:18:26.061583 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.061589 5953 generic.go:356] "Generic (PLEG): container finished" podID="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" containerID="2f40b0a79469bfb66ca1723dfed6cce23aab6b2e980135a50e11335832775639" exitCode=0 Nov 04 03:18:26.062152 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.061597 5953 generic.go:356] "Generic (PLEG): container finished" podID="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" containerID="8df5edc17acbce219b8ee1e04527c0a07bfa8e3ef8772a07acb7c46bbc69b457" exitCode=0 Nov 04 03:18:26.062152 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.061603 5953 generic.go:356] "Generic (PLEG): container finished" podID="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" containerID="5aea6e059cb33894f5559432ed56c5f1f503907e0955d2dfdc7419d997f5dbf8" exitCode=0 Nov 04 03:18:26.062152 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.061608 5953 generic.go:356] "Generic (PLEG): container finished" podID="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" containerID="d5918d555e4e664d0b9dc19f3ea84d5d728df2cb00e5dad2143ea52b72766645" exitCode=0 Nov 04 03:18:26.062152 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.061608 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e","Type":"ContainerDied","Data":"4923b65f9e3377eb7546b564bc8470d59713a6761433c1ccd7fbc667e707250c"} Nov 04 03:18:26.062152 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.061699 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e","Type":"ContainerDied","Data":"dedb1c9816ecad494cbc4bde07bdea706f4a7aaffce47e5e101a9db1a532b062"} Nov 04 03:18:26.062152 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.061641 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.062152 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.061727 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e","Type":"ContainerDied","Data":"2f40b0a79469bfb66ca1723dfed6cce23aab6b2e980135a50e11335832775639"} Nov 04 03:18:26.062152 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.061752 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e","Type":"ContainerDied","Data":"8df5edc17acbce219b8ee1e04527c0a07bfa8e3ef8772a07acb7c46bbc69b457"} Nov 04 03:18:26.062152 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.061776 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e","Type":"ContainerDied","Data":"5aea6e059cb33894f5559432ed56c5f1f503907e0955d2dfdc7419d997f5dbf8"} Nov 04 03:18:26.062152 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.061789 5953 scope.go:122] "RemoveContainer" containerID="4923b65f9e3377eb7546b564bc8470d59713a6761433c1ccd7fbc667e707250c" Nov 04 03:18:26.062152 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.061801 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e","Type":"ContainerDied","Data":"d5918d555e4e664d0b9dc19f3ea84d5d728df2cb00e5dad2143ea52b72766645"} Nov 04 03:18:26.062152 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.061923 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e","Type":"ContainerDied","Data":"791796dc50b8bec54529e842614d63f8768af87659c325aa9ada29014e330c61"} Nov 04 03:18:26.070048 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.070029 5953 scope.go:122] "RemoveContainer" containerID="dedb1c9816ecad494cbc4bde07bdea706f4a7aaffce47e5e101a9db1a532b062" Nov 04 03:18:26.077245 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.077228 5953 scope.go:122] "RemoveContainer" containerID="2f40b0a79469bfb66ca1723dfed6cce23aab6b2e980135a50e11335832775639" Nov 04 03:18:26.084144 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.084128 5953 scope.go:122] "RemoveContainer" containerID="8df5edc17acbce219b8ee1e04527c0a07bfa8e3ef8772a07acb7c46bbc69b457" Nov 04 03:18:26.090768 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.090750 5953 scope.go:122] "RemoveContainer" containerID="5aea6e059cb33894f5559432ed56c5f1f503907e0955d2dfdc7419d997f5dbf8" Nov 04 03:18:26.097826 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.097809 5953 scope.go:122] "RemoveContainer" containerID="d5918d555e4e664d0b9dc19f3ea84d5d728df2cb00e5dad2143ea52b72766645" Nov 04 03:18:26.102951 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.102928 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-thanos-prometheus-http-client-file\") pod \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " Nov 04 03:18:26.103042 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.102967 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-prometheus-k8s-tls\") pod \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " Nov 04 03:18:26.103042 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.103017 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-tls-assets\" (UniqueName: \"kubernetes.io/projected/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-tls-assets\") pod \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " Nov 04 03:18:26.103146 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.103049 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-grpc-tls\") pod \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " Nov 04 03:18:26.103146 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.103083 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-metrics-client-certs\") pod \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " Nov 04 03:18:26.103146 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.103115 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-kube-api-access-9rmmf\" (UniqueName: \"kubernetes.io/projected/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-kube-api-access-9rmmf\") pod \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " Nov 04 03:18:26.103146 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.103141 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-config\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-config\") pod \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " Nov 04 03:18:26.103366 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.103181 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-configmap-serving-certs-ca-bundle\") pod \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " Nov 04 03:18:26.103366 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.103213 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-rulefiles-1\") pod \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " Nov 04 03:18:26.103366 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.103246 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-rulefiles-2\") pod \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " Nov 04 03:18:26.103366 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.103278 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-kube-rbac-proxy\") pod \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " Nov 04 03:18:26.103366 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.103303 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-config-out\" (UniqueName: \"kubernetes.io/empty-dir/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-config-out\") pod \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " Nov 04 03:18:26.103366 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.103324 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-db\") pod \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " Nov 04 03:18:26.103679 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.103371 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " Nov 04 03:18:26.103679 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.103394 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-web-config\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-web-config\") pod \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " Nov 04 03:18:26.103679 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.103435 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-trusted-ca-bundle\") pod \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " Nov 04 03:18:26.103679 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.103462 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-rulefiles-0\") pod \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " Nov 04 03:18:26.103679 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.103490 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-configmap-metrics-client-ca\") pod \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " Nov 04 03:18:26.103679 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.103561 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " Nov 04 03:18:26.103679 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.103649 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-configmap-kubelet-serving-ca-bundle\") pod \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\" (UID: \"d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e\") " Nov 04 03:18:26.104492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.104460 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-configmap-kubelet-serving-ca-bundle" pod "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" (UID: "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:18:26.104843 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.104743 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-rulefiles-2" pod "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" (UID: "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:18:26.105207 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.105066 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-trusted-ca-bundle" pod "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" (UID: "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:18:26.105207 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.105112 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-configmap-metrics-client-ca" pod "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" (UID: "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:18:26.106259 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.106097 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-rulefiles-1" pod "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" (UID: "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:18:26.106680 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.106653 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-db" pod "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" (UID: "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:18:26.106771 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.106743 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-configmap-serving-certs-ca-bundle" pod "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" (UID: "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:18:26.106984 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.106956 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-tls-assets" pod "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" (UID: "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:18:26.107093 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.107074 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-grpc-tls" pod "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" (UID: "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:18:26.107238 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.107219 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-thanos-prometheus-http-client-file" pod "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" (UID: "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:18:26.108151 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.108118 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-rulefiles-0" pod "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" (UID: "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:18:26.108422 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.108396 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-prometheus-k8s-tls" pod "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" (UID: "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:18:26.109141 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.108955 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-prometheus-k8s-kube-rbac-proxy-web" pod "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" (UID: "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:18:26.109141 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.108981 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-config" pod "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" (UID: "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:18:26.109141 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.109116 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-kube-api-access-9rmmf" pod "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" (UID: "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e"). InnerVolumeSpecName "kube-api-access-9rmmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:18:26.109141 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.109134 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-metrics-client-certs" pod "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" (UID: "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:18:26.109803 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.109782 5953 scope.go:122] "RemoveContainer" containerID="c5b0e0618d7907ea3905d068f6441be77893e0bdeb245dd8dab7618ad3fc1818" Nov 04 03:18:26.110174 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.110156 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-kube-rbac-proxy" pod "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" (UID: "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:18:26.110761 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.110741 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-config-out" pod "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" (UID: "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:18:26.110987 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.110903 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-prometheus-k8s-thanos-sidecar-tls" pod "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" (UID: "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:18:26.120111 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.120087 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-web-config" pod "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" (UID: "d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:18:26.131523 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.131506 5953 scope.go:122] "RemoveContainer" containerID="4923b65f9e3377eb7546b564bc8470d59713a6761433c1ccd7fbc667e707250c" Nov 04 03:18:26.131869 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:18:26.131846 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4923b65f9e3377eb7546b564bc8470d59713a6761433c1ccd7fbc667e707250c\": container with ID starting with 4923b65f9e3377eb7546b564bc8470d59713a6761433c1ccd7fbc667e707250c not found: ID does not exist" containerID="4923b65f9e3377eb7546b564bc8470d59713a6761433c1ccd7fbc667e707250c" Nov 04 03:18:26.131923 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.131881 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4923b65f9e3377eb7546b564bc8470d59713a6761433c1ccd7fbc667e707250c"} err="failed to get container status \"4923b65f9e3377eb7546b564bc8470d59713a6761433c1ccd7fbc667e707250c\": rpc error: code = NotFound desc = could not find container \"4923b65f9e3377eb7546b564bc8470d59713a6761433c1ccd7fbc667e707250c\": container with ID starting with 4923b65f9e3377eb7546b564bc8470d59713a6761433c1ccd7fbc667e707250c not found: ID does not exist" Nov 04 03:18:26.131923 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.131898 5953 scope.go:122] "RemoveContainer" containerID="dedb1c9816ecad494cbc4bde07bdea706f4a7aaffce47e5e101a9db1a532b062" Nov 04 03:18:26.132120 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:18:26.132101 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dedb1c9816ecad494cbc4bde07bdea706f4a7aaffce47e5e101a9db1a532b062\": container with ID starting with dedb1c9816ecad494cbc4bde07bdea706f4a7aaffce47e5e101a9db1a532b062 not found: ID does not exist" containerID="dedb1c9816ecad494cbc4bde07bdea706f4a7aaffce47e5e101a9db1a532b062" Nov 04 03:18:26.132219 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.132121 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dedb1c9816ecad494cbc4bde07bdea706f4a7aaffce47e5e101a9db1a532b062"} err="failed to get container status \"dedb1c9816ecad494cbc4bde07bdea706f4a7aaffce47e5e101a9db1a532b062\": rpc error: code = NotFound desc = could not find container \"dedb1c9816ecad494cbc4bde07bdea706f4a7aaffce47e5e101a9db1a532b062\": container with ID starting with dedb1c9816ecad494cbc4bde07bdea706f4a7aaffce47e5e101a9db1a532b062 not found: ID does not exist" Nov 04 03:18:26.132219 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.132132 5953 scope.go:122] "RemoveContainer" containerID="2f40b0a79469bfb66ca1723dfed6cce23aab6b2e980135a50e11335832775639" Nov 04 03:18:26.132416 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:18:26.132397 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f40b0a79469bfb66ca1723dfed6cce23aab6b2e980135a50e11335832775639\": container with ID starting with 2f40b0a79469bfb66ca1723dfed6cce23aab6b2e980135a50e11335832775639 not found: ID does not exist" containerID="2f40b0a79469bfb66ca1723dfed6cce23aab6b2e980135a50e11335832775639" Nov 04 03:18:26.132458 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.132423 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f40b0a79469bfb66ca1723dfed6cce23aab6b2e980135a50e11335832775639"} err="failed to get container status \"2f40b0a79469bfb66ca1723dfed6cce23aab6b2e980135a50e11335832775639\": rpc error: code = NotFound desc = could not find container \"2f40b0a79469bfb66ca1723dfed6cce23aab6b2e980135a50e11335832775639\": container with ID starting with 2f40b0a79469bfb66ca1723dfed6cce23aab6b2e980135a50e11335832775639 not found: ID does not exist" Nov 04 03:18:26.132458 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.132440 5953 scope.go:122] "RemoveContainer" containerID="8df5edc17acbce219b8ee1e04527c0a07bfa8e3ef8772a07acb7c46bbc69b457" Nov 04 03:18:26.132729 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:18:26.132698 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8df5edc17acbce219b8ee1e04527c0a07bfa8e3ef8772a07acb7c46bbc69b457\": container with ID starting with 8df5edc17acbce219b8ee1e04527c0a07bfa8e3ef8772a07acb7c46bbc69b457 not found: ID does not exist" containerID="8df5edc17acbce219b8ee1e04527c0a07bfa8e3ef8772a07acb7c46bbc69b457" Nov 04 03:18:26.132811 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.132733 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8df5edc17acbce219b8ee1e04527c0a07bfa8e3ef8772a07acb7c46bbc69b457"} err="failed to get container status \"8df5edc17acbce219b8ee1e04527c0a07bfa8e3ef8772a07acb7c46bbc69b457\": rpc error: code = NotFound desc = could not find container \"8df5edc17acbce219b8ee1e04527c0a07bfa8e3ef8772a07acb7c46bbc69b457\": container with ID starting with 8df5edc17acbce219b8ee1e04527c0a07bfa8e3ef8772a07acb7c46bbc69b457 not found: ID does not exist" Nov 04 03:18:26.132811 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.132747 5953 scope.go:122] "RemoveContainer" containerID="5aea6e059cb33894f5559432ed56c5f1f503907e0955d2dfdc7419d997f5dbf8" Nov 04 03:18:26.133008 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:18:26.132994 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aea6e059cb33894f5559432ed56c5f1f503907e0955d2dfdc7419d997f5dbf8\": container with ID starting with 5aea6e059cb33894f5559432ed56c5f1f503907e0955d2dfdc7419d997f5dbf8 not found: ID does not exist" containerID="5aea6e059cb33894f5559432ed56c5f1f503907e0955d2dfdc7419d997f5dbf8" Nov 04 03:18:26.133053 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.133012 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aea6e059cb33894f5559432ed56c5f1f503907e0955d2dfdc7419d997f5dbf8"} err="failed to get container status \"5aea6e059cb33894f5559432ed56c5f1f503907e0955d2dfdc7419d997f5dbf8\": rpc error: code = NotFound desc = could not find container \"5aea6e059cb33894f5559432ed56c5f1f503907e0955d2dfdc7419d997f5dbf8\": container with ID starting with 5aea6e059cb33894f5559432ed56c5f1f503907e0955d2dfdc7419d997f5dbf8 not found: ID does not exist" Nov 04 03:18:26.133053 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.133026 5953 scope.go:122] "RemoveContainer" containerID="d5918d555e4e664d0b9dc19f3ea84d5d728df2cb00e5dad2143ea52b72766645" Nov 04 03:18:26.133296 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:18:26.133280 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5918d555e4e664d0b9dc19f3ea84d5d728df2cb00e5dad2143ea52b72766645\": container with ID starting with d5918d555e4e664d0b9dc19f3ea84d5d728df2cb00e5dad2143ea52b72766645 not found: ID does not exist" containerID="d5918d555e4e664d0b9dc19f3ea84d5d728df2cb00e5dad2143ea52b72766645" Nov 04 03:18:26.133337 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.133300 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5918d555e4e664d0b9dc19f3ea84d5d728df2cb00e5dad2143ea52b72766645"} err="failed to get container status \"d5918d555e4e664d0b9dc19f3ea84d5d728df2cb00e5dad2143ea52b72766645\": rpc error: code = NotFound desc = could not find container \"d5918d555e4e664d0b9dc19f3ea84d5d728df2cb00e5dad2143ea52b72766645\": container with ID starting with d5918d555e4e664d0b9dc19f3ea84d5d728df2cb00e5dad2143ea52b72766645 not found: ID does not exist" Nov 04 03:18:26.133337 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.133314 5953 scope.go:122] "RemoveContainer" containerID="c5b0e0618d7907ea3905d068f6441be77893e0bdeb245dd8dab7618ad3fc1818" Nov 04 03:18:26.133582 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:18:26.133553 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5b0e0618d7907ea3905d068f6441be77893e0bdeb245dd8dab7618ad3fc1818\": container with ID starting with c5b0e0618d7907ea3905d068f6441be77893e0bdeb245dd8dab7618ad3fc1818 not found: ID does not exist" containerID="c5b0e0618d7907ea3905d068f6441be77893e0bdeb245dd8dab7618ad3fc1818" Nov 04 03:18:26.133674 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.133580 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5b0e0618d7907ea3905d068f6441be77893e0bdeb245dd8dab7618ad3fc1818"} err="failed to get container status \"c5b0e0618d7907ea3905d068f6441be77893e0bdeb245dd8dab7618ad3fc1818\": rpc error: code = NotFound desc = could not find container \"c5b0e0618d7907ea3905d068f6441be77893e0bdeb245dd8dab7618ad3fc1818\": container with ID starting with c5b0e0618d7907ea3905d068f6441be77893e0bdeb245dd8dab7618ad3fc1818 not found: ID does not exist" Nov 04 03:18:26.133674 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.133600 5953 scope.go:122] "RemoveContainer" containerID="4923b65f9e3377eb7546b564bc8470d59713a6761433c1ccd7fbc667e707250c" Nov 04 03:18:26.133887 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.133859 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4923b65f9e3377eb7546b564bc8470d59713a6761433c1ccd7fbc667e707250c"} err="failed to get container status \"4923b65f9e3377eb7546b564bc8470d59713a6761433c1ccd7fbc667e707250c\": rpc error: code = NotFound desc = could not find container \"4923b65f9e3377eb7546b564bc8470d59713a6761433c1ccd7fbc667e707250c\": container with ID starting with 4923b65f9e3377eb7546b564bc8470d59713a6761433c1ccd7fbc667e707250c not found: ID does not exist" Nov 04 03:18:26.133887 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.133879 5953 scope.go:122] "RemoveContainer" containerID="dedb1c9816ecad494cbc4bde07bdea706f4a7aaffce47e5e101a9db1a532b062" Nov 04 03:18:26.134116 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.134099 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dedb1c9816ecad494cbc4bde07bdea706f4a7aaffce47e5e101a9db1a532b062"} err="failed to get container status \"dedb1c9816ecad494cbc4bde07bdea706f4a7aaffce47e5e101a9db1a532b062\": rpc error: code = NotFound desc = could not find container \"dedb1c9816ecad494cbc4bde07bdea706f4a7aaffce47e5e101a9db1a532b062\": container with ID starting with dedb1c9816ecad494cbc4bde07bdea706f4a7aaffce47e5e101a9db1a532b062 not found: ID does not exist" Nov 04 03:18:26.134159 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.134117 5953 scope.go:122] "RemoveContainer" containerID="2f40b0a79469bfb66ca1723dfed6cce23aab6b2e980135a50e11335832775639" Nov 04 03:18:26.134335 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.134317 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f40b0a79469bfb66ca1723dfed6cce23aab6b2e980135a50e11335832775639"} err="failed to get container status \"2f40b0a79469bfb66ca1723dfed6cce23aab6b2e980135a50e11335832775639\": rpc error: code = NotFound desc = could not find container \"2f40b0a79469bfb66ca1723dfed6cce23aab6b2e980135a50e11335832775639\": container with ID starting with 2f40b0a79469bfb66ca1723dfed6cce23aab6b2e980135a50e11335832775639 not found: ID does not exist" Nov 04 03:18:26.134400 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.134336 5953 scope.go:122] "RemoveContainer" containerID="8df5edc17acbce219b8ee1e04527c0a07bfa8e3ef8772a07acb7c46bbc69b457" Nov 04 03:18:26.134595 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.134578 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8df5edc17acbce219b8ee1e04527c0a07bfa8e3ef8772a07acb7c46bbc69b457"} err="failed to get container status \"8df5edc17acbce219b8ee1e04527c0a07bfa8e3ef8772a07acb7c46bbc69b457\": rpc error: code = NotFound desc = could not find container \"8df5edc17acbce219b8ee1e04527c0a07bfa8e3ef8772a07acb7c46bbc69b457\": container with ID starting with 8df5edc17acbce219b8ee1e04527c0a07bfa8e3ef8772a07acb7c46bbc69b457 not found: ID does not exist" Nov 04 03:18:26.134656 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.134604 5953 scope.go:122] "RemoveContainer" containerID="5aea6e059cb33894f5559432ed56c5f1f503907e0955d2dfdc7419d997f5dbf8" Nov 04 03:18:26.134855 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.134840 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aea6e059cb33894f5559432ed56c5f1f503907e0955d2dfdc7419d997f5dbf8"} err="failed to get container status \"5aea6e059cb33894f5559432ed56c5f1f503907e0955d2dfdc7419d997f5dbf8\": rpc error: code = NotFound desc = could not find container \"5aea6e059cb33894f5559432ed56c5f1f503907e0955d2dfdc7419d997f5dbf8\": container with ID starting with 5aea6e059cb33894f5559432ed56c5f1f503907e0955d2dfdc7419d997f5dbf8 not found: ID does not exist" Nov 04 03:18:26.134895 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.134856 5953 scope.go:122] "RemoveContainer" containerID="d5918d555e4e664d0b9dc19f3ea84d5d728df2cb00e5dad2143ea52b72766645" Nov 04 03:18:26.135085 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.135067 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5918d555e4e664d0b9dc19f3ea84d5d728df2cb00e5dad2143ea52b72766645"} err="failed to get container status \"d5918d555e4e664d0b9dc19f3ea84d5d728df2cb00e5dad2143ea52b72766645\": rpc error: code = NotFound desc = could not find container \"d5918d555e4e664d0b9dc19f3ea84d5d728df2cb00e5dad2143ea52b72766645\": container with ID starting with d5918d555e4e664d0b9dc19f3ea84d5d728df2cb00e5dad2143ea52b72766645 not found: ID does not exist" Nov 04 03:18:26.135150 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.135086 5953 scope.go:122] "RemoveContainer" containerID="c5b0e0618d7907ea3905d068f6441be77893e0bdeb245dd8dab7618ad3fc1818" Nov 04 03:18:26.135323 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.135305 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5b0e0618d7907ea3905d068f6441be77893e0bdeb245dd8dab7618ad3fc1818"} err="failed to get container status \"c5b0e0618d7907ea3905d068f6441be77893e0bdeb245dd8dab7618ad3fc1818\": rpc error: code = NotFound desc = could not find container \"c5b0e0618d7907ea3905d068f6441be77893e0bdeb245dd8dab7618ad3fc1818\": container with ID starting with c5b0e0618d7907ea3905d068f6441be77893e0bdeb245dd8dab7618ad3fc1818 not found: ID does not exist" Nov 04 03:18:26.135378 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.135324 5953 scope.go:122] "RemoveContainer" containerID="4923b65f9e3377eb7546b564bc8470d59713a6761433c1ccd7fbc667e707250c" Nov 04 03:18:26.135526 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.135505 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4923b65f9e3377eb7546b564bc8470d59713a6761433c1ccd7fbc667e707250c"} err="failed to get container status \"4923b65f9e3377eb7546b564bc8470d59713a6761433c1ccd7fbc667e707250c\": rpc error: code = NotFound desc = could not find container \"4923b65f9e3377eb7546b564bc8470d59713a6761433c1ccd7fbc667e707250c\": container with ID starting with 4923b65f9e3377eb7546b564bc8470d59713a6761433c1ccd7fbc667e707250c not found: ID does not exist" Nov 04 03:18:26.135567 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.135527 5953 scope.go:122] "RemoveContainer" containerID="dedb1c9816ecad494cbc4bde07bdea706f4a7aaffce47e5e101a9db1a532b062" Nov 04 03:18:26.135744 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.135724 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dedb1c9816ecad494cbc4bde07bdea706f4a7aaffce47e5e101a9db1a532b062"} err="failed to get container status \"dedb1c9816ecad494cbc4bde07bdea706f4a7aaffce47e5e101a9db1a532b062\": rpc error: code = NotFound desc = could not find container \"dedb1c9816ecad494cbc4bde07bdea706f4a7aaffce47e5e101a9db1a532b062\": container with ID starting with dedb1c9816ecad494cbc4bde07bdea706f4a7aaffce47e5e101a9db1a532b062 not found: ID does not exist" Nov 04 03:18:26.135791 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.135743 5953 scope.go:122] "RemoveContainer" containerID="2f40b0a79469bfb66ca1723dfed6cce23aab6b2e980135a50e11335832775639" Nov 04 03:18:26.135994 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.135976 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f40b0a79469bfb66ca1723dfed6cce23aab6b2e980135a50e11335832775639"} err="failed to get container status \"2f40b0a79469bfb66ca1723dfed6cce23aab6b2e980135a50e11335832775639\": rpc error: code = NotFound desc = could not find container \"2f40b0a79469bfb66ca1723dfed6cce23aab6b2e980135a50e11335832775639\": container with ID starting with 2f40b0a79469bfb66ca1723dfed6cce23aab6b2e980135a50e11335832775639 not found: ID does not exist" Nov 04 03:18:26.136062 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.135995 5953 scope.go:122] "RemoveContainer" containerID="8df5edc17acbce219b8ee1e04527c0a07bfa8e3ef8772a07acb7c46bbc69b457" Nov 04 03:18:26.136237 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.136216 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8df5edc17acbce219b8ee1e04527c0a07bfa8e3ef8772a07acb7c46bbc69b457"} err="failed to get container status \"8df5edc17acbce219b8ee1e04527c0a07bfa8e3ef8772a07acb7c46bbc69b457\": rpc error: code = NotFound desc = could not find container \"8df5edc17acbce219b8ee1e04527c0a07bfa8e3ef8772a07acb7c46bbc69b457\": container with ID starting with 8df5edc17acbce219b8ee1e04527c0a07bfa8e3ef8772a07acb7c46bbc69b457 not found: ID does not exist" Nov 04 03:18:26.136237 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.136236 5953 scope.go:122] "RemoveContainer" containerID="5aea6e059cb33894f5559432ed56c5f1f503907e0955d2dfdc7419d997f5dbf8" Nov 04 03:18:26.136477 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.136461 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aea6e059cb33894f5559432ed56c5f1f503907e0955d2dfdc7419d997f5dbf8"} err="failed to get container status \"5aea6e059cb33894f5559432ed56c5f1f503907e0955d2dfdc7419d997f5dbf8\": rpc error: code = NotFound desc = could not find container \"5aea6e059cb33894f5559432ed56c5f1f503907e0955d2dfdc7419d997f5dbf8\": container with ID starting with 5aea6e059cb33894f5559432ed56c5f1f503907e0955d2dfdc7419d997f5dbf8 not found: ID does not exist" Nov 04 03:18:26.136477 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.136476 5953 scope.go:122] "RemoveContainer" containerID="d5918d555e4e664d0b9dc19f3ea84d5d728df2cb00e5dad2143ea52b72766645" Nov 04 03:18:26.136710 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.136693 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5918d555e4e664d0b9dc19f3ea84d5d728df2cb00e5dad2143ea52b72766645"} err="failed to get container status \"d5918d555e4e664d0b9dc19f3ea84d5d728df2cb00e5dad2143ea52b72766645\": rpc error: code = NotFound desc = could not find container \"d5918d555e4e664d0b9dc19f3ea84d5d728df2cb00e5dad2143ea52b72766645\": container with ID starting with d5918d555e4e664d0b9dc19f3ea84d5d728df2cb00e5dad2143ea52b72766645 not found: ID does not exist" Nov 04 03:18:26.136710 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.136709 5953 scope.go:122] "RemoveContainer" containerID="c5b0e0618d7907ea3905d068f6441be77893e0bdeb245dd8dab7618ad3fc1818" Nov 04 03:18:26.136927 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.136911 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5b0e0618d7907ea3905d068f6441be77893e0bdeb245dd8dab7618ad3fc1818"} err="failed to get container status \"c5b0e0618d7907ea3905d068f6441be77893e0bdeb245dd8dab7618ad3fc1818\": rpc error: code = NotFound desc = could not find container \"c5b0e0618d7907ea3905d068f6441be77893e0bdeb245dd8dab7618ad3fc1818\": container with ID starting with c5b0e0618d7907ea3905d068f6441be77893e0bdeb245dd8dab7618ad3fc1818 not found: ID does not exist" Nov 04 03:18:26.136976 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.136928 5953 scope.go:122] "RemoveContainer" containerID="4923b65f9e3377eb7546b564bc8470d59713a6761433c1ccd7fbc667e707250c" Nov 04 03:18:26.137179 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.137161 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4923b65f9e3377eb7546b564bc8470d59713a6761433c1ccd7fbc667e707250c"} err="failed to get container status \"4923b65f9e3377eb7546b564bc8470d59713a6761433c1ccd7fbc667e707250c\": rpc error: code = NotFound desc = could not find container \"4923b65f9e3377eb7546b564bc8470d59713a6761433c1ccd7fbc667e707250c\": container with ID starting with 4923b65f9e3377eb7546b564bc8470d59713a6761433c1ccd7fbc667e707250c not found: ID does not exist" Nov 04 03:18:26.137179 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.137178 5953 scope.go:122] "RemoveContainer" containerID="dedb1c9816ecad494cbc4bde07bdea706f4a7aaffce47e5e101a9db1a532b062" Nov 04 03:18:26.137407 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.137388 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dedb1c9816ecad494cbc4bde07bdea706f4a7aaffce47e5e101a9db1a532b062"} err="failed to get container status \"dedb1c9816ecad494cbc4bde07bdea706f4a7aaffce47e5e101a9db1a532b062\": rpc error: code = NotFound desc = could not find container \"dedb1c9816ecad494cbc4bde07bdea706f4a7aaffce47e5e101a9db1a532b062\": container with ID starting with dedb1c9816ecad494cbc4bde07bdea706f4a7aaffce47e5e101a9db1a532b062 not found: ID does not exist" Nov 04 03:18:26.137456 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.137407 5953 scope.go:122] "RemoveContainer" containerID="2f40b0a79469bfb66ca1723dfed6cce23aab6b2e980135a50e11335832775639" Nov 04 03:18:26.137669 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.137654 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f40b0a79469bfb66ca1723dfed6cce23aab6b2e980135a50e11335832775639"} err="failed to get container status \"2f40b0a79469bfb66ca1723dfed6cce23aab6b2e980135a50e11335832775639\": rpc error: code = NotFound desc = could not find container \"2f40b0a79469bfb66ca1723dfed6cce23aab6b2e980135a50e11335832775639\": container with ID starting with 2f40b0a79469bfb66ca1723dfed6cce23aab6b2e980135a50e11335832775639 not found: ID does not exist" Nov 04 03:18:26.137709 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.137669 5953 scope.go:122] "RemoveContainer" containerID="8df5edc17acbce219b8ee1e04527c0a07bfa8e3ef8772a07acb7c46bbc69b457" Nov 04 03:18:26.137880 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.137861 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8df5edc17acbce219b8ee1e04527c0a07bfa8e3ef8772a07acb7c46bbc69b457"} err="failed to get container status \"8df5edc17acbce219b8ee1e04527c0a07bfa8e3ef8772a07acb7c46bbc69b457\": rpc error: code = NotFound desc = could not find container \"8df5edc17acbce219b8ee1e04527c0a07bfa8e3ef8772a07acb7c46bbc69b457\": container with ID starting with 8df5edc17acbce219b8ee1e04527c0a07bfa8e3ef8772a07acb7c46bbc69b457 not found: ID does not exist" Nov 04 03:18:26.137919 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.137881 5953 scope.go:122] "RemoveContainer" containerID="5aea6e059cb33894f5559432ed56c5f1f503907e0955d2dfdc7419d997f5dbf8" Nov 04 03:18:26.138110 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.138092 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aea6e059cb33894f5559432ed56c5f1f503907e0955d2dfdc7419d997f5dbf8"} err="failed to get container status \"5aea6e059cb33894f5559432ed56c5f1f503907e0955d2dfdc7419d997f5dbf8\": rpc error: code = NotFound desc = could not find container \"5aea6e059cb33894f5559432ed56c5f1f503907e0955d2dfdc7419d997f5dbf8\": container with ID starting with 5aea6e059cb33894f5559432ed56c5f1f503907e0955d2dfdc7419d997f5dbf8 not found: ID does not exist" Nov 04 03:18:26.138152 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.138111 5953 scope.go:122] "RemoveContainer" containerID="d5918d555e4e664d0b9dc19f3ea84d5d728df2cb00e5dad2143ea52b72766645" Nov 04 03:18:26.138323 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.138304 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5918d555e4e664d0b9dc19f3ea84d5d728df2cb00e5dad2143ea52b72766645"} err="failed to get container status \"d5918d555e4e664d0b9dc19f3ea84d5d728df2cb00e5dad2143ea52b72766645\": rpc error: code = NotFound desc = could not find container \"d5918d555e4e664d0b9dc19f3ea84d5d728df2cb00e5dad2143ea52b72766645\": container with ID starting with d5918d555e4e664d0b9dc19f3ea84d5d728df2cb00e5dad2143ea52b72766645 not found: ID does not exist" Nov 04 03:18:26.138363 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.138324 5953 scope.go:122] "RemoveContainer" containerID="c5b0e0618d7907ea3905d068f6441be77893e0bdeb245dd8dab7618ad3fc1818" Nov 04 03:18:26.138574 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.138557 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5b0e0618d7907ea3905d068f6441be77893e0bdeb245dd8dab7618ad3fc1818"} err="failed to get container status \"c5b0e0618d7907ea3905d068f6441be77893e0bdeb245dd8dab7618ad3fc1818\": rpc error: code = NotFound desc = could not find container \"c5b0e0618d7907ea3905d068f6441be77893e0bdeb245dd8dab7618ad3fc1818\": container with ID starting with c5b0e0618d7907ea3905d068f6441be77893e0bdeb245dd8dab7618ad3fc1818 not found: ID does not exist" Nov 04 03:18:26.138632 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.138575 5953 scope.go:122] "RemoveContainer" containerID="4923b65f9e3377eb7546b564bc8470d59713a6761433c1ccd7fbc667e707250c" Nov 04 03:18:26.138900 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.138883 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4923b65f9e3377eb7546b564bc8470d59713a6761433c1ccd7fbc667e707250c"} err="failed to get container status \"4923b65f9e3377eb7546b564bc8470d59713a6761433c1ccd7fbc667e707250c\": rpc error: code = NotFound desc = could not find container \"4923b65f9e3377eb7546b564bc8470d59713a6761433c1ccd7fbc667e707250c\": container with ID starting with 4923b65f9e3377eb7546b564bc8470d59713a6761433c1ccd7fbc667e707250c not found: ID does not exist" Nov 04 03:18:26.138959 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.138900 5953 scope.go:122] "RemoveContainer" containerID="dedb1c9816ecad494cbc4bde07bdea706f4a7aaffce47e5e101a9db1a532b062" Nov 04 03:18:26.139131 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.139113 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dedb1c9816ecad494cbc4bde07bdea706f4a7aaffce47e5e101a9db1a532b062"} err="failed to get container status \"dedb1c9816ecad494cbc4bde07bdea706f4a7aaffce47e5e101a9db1a532b062\": rpc error: code = NotFound desc = could not find container \"dedb1c9816ecad494cbc4bde07bdea706f4a7aaffce47e5e101a9db1a532b062\": container with ID starting with dedb1c9816ecad494cbc4bde07bdea706f4a7aaffce47e5e101a9db1a532b062 not found: ID does not exist" Nov 04 03:18:26.139174 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.139132 5953 scope.go:122] "RemoveContainer" containerID="2f40b0a79469bfb66ca1723dfed6cce23aab6b2e980135a50e11335832775639" Nov 04 03:18:26.139341 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.139326 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f40b0a79469bfb66ca1723dfed6cce23aab6b2e980135a50e11335832775639"} err="failed to get container status \"2f40b0a79469bfb66ca1723dfed6cce23aab6b2e980135a50e11335832775639\": rpc error: code = NotFound desc = could not find container \"2f40b0a79469bfb66ca1723dfed6cce23aab6b2e980135a50e11335832775639\": container with ID starting with 2f40b0a79469bfb66ca1723dfed6cce23aab6b2e980135a50e11335832775639 not found: ID does not exist" Nov 04 03:18:26.139393 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.139342 5953 scope.go:122] "RemoveContainer" containerID="8df5edc17acbce219b8ee1e04527c0a07bfa8e3ef8772a07acb7c46bbc69b457" Nov 04 03:18:26.139581 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.139565 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8df5edc17acbce219b8ee1e04527c0a07bfa8e3ef8772a07acb7c46bbc69b457"} err="failed to get container status \"8df5edc17acbce219b8ee1e04527c0a07bfa8e3ef8772a07acb7c46bbc69b457\": rpc error: code = NotFound desc = could not find container \"8df5edc17acbce219b8ee1e04527c0a07bfa8e3ef8772a07acb7c46bbc69b457\": container with ID starting with 8df5edc17acbce219b8ee1e04527c0a07bfa8e3ef8772a07acb7c46bbc69b457 not found: ID does not exist" Nov 04 03:18:26.139634 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.139582 5953 scope.go:122] "RemoveContainer" containerID="5aea6e059cb33894f5559432ed56c5f1f503907e0955d2dfdc7419d997f5dbf8" Nov 04 03:18:26.139832 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.139817 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aea6e059cb33894f5559432ed56c5f1f503907e0955d2dfdc7419d997f5dbf8"} err="failed to get container status \"5aea6e059cb33894f5559432ed56c5f1f503907e0955d2dfdc7419d997f5dbf8\": rpc error: code = NotFound desc = could not find container \"5aea6e059cb33894f5559432ed56c5f1f503907e0955d2dfdc7419d997f5dbf8\": container with ID starting with 5aea6e059cb33894f5559432ed56c5f1f503907e0955d2dfdc7419d997f5dbf8 not found: ID does not exist" Nov 04 03:18:26.139881 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.139834 5953 scope.go:122] "RemoveContainer" containerID="d5918d555e4e664d0b9dc19f3ea84d5d728df2cb00e5dad2143ea52b72766645" Nov 04 03:18:26.140125 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.140109 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5918d555e4e664d0b9dc19f3ea84d5d728df2cb00e5dad2143ea52b72766645"} err="failed to get container status \"d5918d555e4e664d0b9dc19f3ea84d5d728df2cb00e5dad2143ea52b72766645\": rpc error: code = NotFound desc = could not find container \"d5918d555e4e664d0b9dc19f3ea84d5d728df2cb00e5dad2143ea52b72766645\": container with ID starting with d5918d555e4e664d0b9dc19f3ea84d5d728df2cb00e5dad2143ea52b72766645 not found: ID does not exist" Nov 04 03:18:26.140166 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.140126 5953 scope.go:122] "RemoveContainer" containerID="c5b0e0618d7907ea3905d068f6441be77893e0bdeb245dd8dab7618ad3fc1818" Nov 04 03:18:26.140346 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.140318 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5b0e0618d7907ea3905d068f6441be77893e0bdeb245dd8dab7618ad3fc1818"} err="failed to get container status \"c5b0e0618d7907ea3905d068f6441be77893e0bdeb245dd8dab7618ad3fc1818\": rpc error: code = NotFound desc = could not find container \"c5b0e0618d7907ea3905d068f6441be77893e0bdeb245dd8dab7618ad3fc1818\": container with ID starting with c5b0e0618d7907ea3905d068f6441be77893e0bdeb245dd8dab7618ad3fc1818 not found: ID does not exist" Nov 04 03:18:26.140346 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.140338 5953 scope.go:122] "RemoveContainer" containerID="4923b65f9e3377eb7546b564bc8470d59713a6761433c1ccd7fbc667e707250c" Nov 04 03:18:26.140568 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.140547 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4923b65f9e3377eb7546b564bc8470d59713a6761433c1ccd7fbc667e707250c"} err="failed to get container status \"4923b65f9e3377eb7546b564bc8470d59713a6761433c1ccd7fbc667e707250c\": rpc error: code = NotFound desc = could not find container \"4923b65f9e3377eb7546b564bc8470d59713a6761433c1ccd7fbc667e707250c\": container with ID starting with 4923b65f9e3377eb7546b564bc8470d59713a6761433c1ccd7fbc667e707250c not found: ID does not exist" Nov 04 03:18:26.140568 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.140567 5953 scope.go:122] "RemoveContainer" containerID="dedb1c9816ecad494cbc4bde07bdea706f4a7aaffce47e5e101a9db1a532b062" Nov 04 03:18:26.140807 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.140781 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dedb1c9816ecad494cbc4bde07bdea706f4a7aaffce47e5e101a9db1a532b062"} err="failed to get container status \"dedb1c9816ecad494cbc4bde07bdea706f4a7aaffce47e5e101a9db1a532b062\": rpc error: code = NotFound desc = could not find container \"dedb1c9816ecad494cbc4bde07bdea706f4a7aaffce47e5e101a9db1a532b062\": container with ID starting with dedb1c9816ecad494cbc4bde07bdea706f4a7aaffce47e5e101a9db1a532b062 not found: ID does not exist" Nov 04 03:18:26.140856 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.140807 5953 scope.go:122] "RemoveContainer" containerID="2f40b0a79469bfb66ca1723dfed6cce23aab6b2e980135a50e11335832775639" Nov 04 03:18:26.141045 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.141030 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f40b0a79469bfb66ca1723dfed6cce23aab6b2e980135a50e11335832775639"} err="failed to get container status \"2f40b0a79469bfb66ca1723dfed6cce23aab6b2e980135a50e11335832775639\": rpc error: code = NotFound desc = could not find container \"2f40b0a79469bfb66ca1723dfed6cce23aab6b2e980135a50e11335832775639\": container with ID starting with 2f40b0a79469bfb66ca1723dfed6cce23aab6b2e980135a50e11335832775639 not found: ID does not exist" Nov 04 03:18:26.141102 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.141046 5953 scope.go:122] "RemoveContainer" containerID="8df5edc17acbce219b8ee1e04527c0a07bfa8e3ef8772a07acb7c46bbc69b457" Nov 04 03:18:26.141279 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.141262 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8df5edc17acbce219b8ee1e04527c0a07bfa8e3ef8772a07acb7c46bbc69b457"} err="failed to get container status \"8df5edc17acbce219b8ee1e04527c0a07bfa8e3ef8772a07acb7c46bbc69b457\": rpc error: code = NotFound desc = could not find container \"8df5edc17acbce219b8ee1e04527c0a07bfa8e3ef8772a07acb7c46bbc69b457\": container with ID starting with 8df5edc17acbce219b8ee1e04527c0a07bfa8e3ef8772a07acb7c46bbc69b457 not found: ID does not exist" Nov 04 03:18:26.141321 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.141280 5953 scope.go:122] "RemoveContainer" containerID="5aea6e059cb33894f5559432ed56c5f1f503907e0955d2dfdc7419d997f5dbf8" Nov 04 03:18:26.141498 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.141484 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aea6e059cb33894f5559432ed56c5f1f503907e0955d2dfdc7419d997f5dbf8"} err="failed to get container status \"5aea6e059cb33894f5559432ed56c5f1f503907e0955d2dfdc7419d997f5dbf8\": rpc error: code = NotFound desc = could not find container \"5aea6e059cb33894f5559432ed56c5f1f503907e0955d2dfdc7419d997f5dbf8\": container with ID starting with 5aea6e059cb33894f5559432ed56c5f1f503907e0955d2dfdc7419d997f5dbf8 not found: ID does not exist" Nov 04 03:18:26.141535 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.141497 5953 scope.go:122] "RemoveContainer" containerID="d5918d555e4e664d0b9dc19f3ea84d5d728df2cb00e5dad2143ea52b72766645" Nov 04 03:18:26.141751 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.141733 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5918d555e4e664d0b9dc19f3ea84d5d728df2cb00e5dad2143ea52b72766645"} err="failed to get container status \"d5918d555e4e664d0b9dc19f3ea84d5d728df2cb00e5dad2143ea52b72766645\": rpc error: code = NotFound desc = could not find container \"d5918d555e4e664d0b9dc19f3ea84d5d728df2cb00e5dad2143ea52b72766645\": container with ID starting with d5918d555e4e664d0b9dc19f3ea84d5d728df2cb00e5dad2143ea52b72766645 not found: ID does not exist" Nov 04 03:18:26.141798 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.141752 5953 scope.go:122] "RemoveContainer" containerID="c5b0e0618d7907ea3905d068f6441be77893e0bdeb245dd8dab7618ad3fc1818" Nov 04 03:18:26.141993 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.141976 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5b0e0618d7907ea3905d068f6441be77893e0bdeb245dd8dab7618ad3fc1818"} err="failed to get container status \"c5b0e0618d7907ea3905d068f6441be77893e0bdeb245dd8dab7618ad3fc1818\": rpc error: code = NotFound desc = could not find container \"c5b0e0618d7907ea3905d068f6441be77893e0bdeb245dd8dab7618ad3fc1818\": container with ID starting with c5b0e0618d7907ea3905d068f6441be77893e0bdeb245dd8dab7618ad3fc1818 not found: ID does not exist" Nov 04 03:18:26.204961 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.204863 5953 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:26.204961 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.204905 5953 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:26.204961 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.204924 5953 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-tls-assets\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:26.204961 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.204960 5953 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-grpc-tls\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:26.205229 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.204992 5953 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-metrics-client-certs\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:26.205229 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.205010 5953 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rmmf\" (UniqueName: \"kubernetes.io/projected/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-kube-api-access-9rmmf\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:26.205229 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.205028 5953 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-config\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:26.205229 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.205042 5953 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:26.205229 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.205056 5953 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:26.205229 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.205069 5953 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:26.205229 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.205079 5953 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-kube-rbac-proxy\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:26.205229 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.205088 5953 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-config-out\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:26.205229 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.205097 5953 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-db\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:26.205229 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.205106 5953 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:26.205229 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.205118 5953 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-web-config\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:26.205229 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.205131 5953 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:26.205229 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.205140 5953 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:26.205229 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.205148 5953 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-configmap-metrics-client-ca\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:26.205229 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.205157 5953 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:26.205229 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.205166 5953 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:26.484266 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.484231 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 03:18:26.488758 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.488731 5953 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 03:18:26.534388 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.534357 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 03:18:26.534677 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.534665 5953 cpu_manager.go:414] "containerMap: removing container" podUID="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" containerName="thanos-sidecar" Nov 04 03:18:26.534738 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.534678 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" containerName="thanos-sidecar" Nov 04 03:18:26.534738 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.534694 5953 cpu_manager.go:414] "containerMap: removing container" podUID="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" containerName="kube-rbac-proxy-thanos" Nov 04 03:18:26.534738 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.534699 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" containerName="kube-rbac-proxy-thanos" Nov 04 03:18:26.534738 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.534708 5953 cpu_manager.go:414] "containerMap: removing container" podUID="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" containerName="kube-rbac-proxy-web" Nov 04 03:18:26.534738 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.534714 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" containerName="kube-rbac-proxy-web" Nov 04 03:18:26.534738 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.534720 5953 cpu_manager.go:414] "containerMap: removing container" podUID="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" containerName="config-reloader" Nov 04 03:18:26.534738 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.534726 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" containerName="config-reloader" Nov 04 03:18:26.534938 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.534742 5953 cpu_manager.go:414] "containerMap: removing container" podUID="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" containerName="kube-rbac-proxy" Nov 04 03:18:26.534938 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.534747 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" containerName="kube-rbac-proxy" Nov 04 03:18:26.534938 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.534754 5953 cpu_manager.go:414] "containerMap: removing container" podUID="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" containerName="init-config-reloader" Nov 04 03:18:26.534938 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.534758 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" containerName="init-config-reloader" Nov 04 03:18:26.534938 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.534767 5953 cpu_manager.go:414] "containerMap: removing container" podUID="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" containerName="prometheus" Nov 04 03:18:26.534938 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.534771 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" containerName="prometheus" Nov 04 03:18:26.534938 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.534815 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" containerName="kube-rbac-proxy-thanos" Nov 04 03:18:26.534938 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.534821 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" containerName="kube-rbac-proxy-web" Nov 04 03:18:26.534938 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.534827 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" containerName="config-reloader" Nov 04 03:18:26.534938 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.534834 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" containerName="kube-rbac-proxy" Nov 04 03:18:26.534938 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.534839 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" containerName="init-config-reloader" Nov 04 03:18:26.534938 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.534844 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" containerName="prometheus" Nov 04 03:18:26.534938 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.534851 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" containerName="thanos-sidecar" Nov 04 03:18:26.538219 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.538196 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.592427 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.592208 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 04 03:18:26.592427 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.592264 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 04 03:18:26.592427 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.592291 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 04 03:18:26.592793 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.592771 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 03:18:26.646342 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.643557 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 04 03:18:26.646342 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.643714 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 04 03:18:26.646342 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.643850 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 04 03:18:26.646342 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.643932 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 04 03:18:26.646342 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.644131 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 04 03:18:26.646342 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.644212 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 04 03:18:26.646342 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.644285 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 04 03:18:26.646342 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.644515 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-bh6gw\"" Nov 04 03:18:26.646342 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.644639 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 04 03:18:26.646342 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.646208 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e4spppbfh1spp\"" Nov 04 03:18:26.646884 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.646377 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 04 03:18:26.646884 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.646506 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 04 03:18:26.646884 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.646563 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 04 03:18:26.709890 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.709854 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e7a9993d-e950-4dd4-af7c-4333701ffb24-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.710043 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.709895 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7a9993d-e950-4dd4-af7c-4333701ffb24-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.710043 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.709921 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7a9993d-e950-4dd4-af7c-4333701ffb24-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.710126 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.710062 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e7a9993d-e950-4dd4-af7c-4333701ffb24-config\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.710126 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.710096 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e7a9993d-e950-4dd4-af7c-4333701ffb24-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.710207 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.710130 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e7a9993d-e950-4dd4-af7c-4333701ffb24-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.710207 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.710181 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e7a9993d-e950-4dd4-af7c-4333701ffb24-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.710301 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.710279 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e7a9993d-e950-4dd4-af7c-4333701ffb24-config-out\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.710354 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.710320 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e7a9993d-e950-4dd4-af7c-4333701ffb24-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.710397 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.710360 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7a9993d-e950-4dd4-af7c-4333701ffb24-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.710397 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.710380 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e7a9993d-e950-4dd4-af7c-4333701ffb24-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.710491 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.710395 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e7a9993d-e950-4dd4-af7c-4333701ffb24-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.710491 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.710439 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e7a9993d-e950-4dd4-af7c-4333701ffb24-web-config\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.710597 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.710492 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wml7\" (UniqueName: \"kubernetes.io/projected/e7a9993d-e950-4dd4-af7c-4333701ffb24-kube-api-access-2wml7\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.710597 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.710536 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e7a9993d-e950-4dd4-af7c-4333701ffb24-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.710597 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.710561 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e7a9993d-e950-4dd4-af7c-4333701ffb24-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.710597 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.710585 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e7a9993d-e950-4dd4-af7c-4333701ffb24-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.710791 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.710638 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7a9993d-e950-4dd4-af7c-4333701ffb24-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.710791 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.710665 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e7a9993d-e950-4dd4-af7c-4333701ffb24-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.710791 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.710692 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e7a9993d-e950-4dd4-af7c-4333701ffb24-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.811968 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.811923 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e7a9993d-e950-4dd4-af7c-4333701ffb24-config\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.811968 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.811974 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e7a9993d-e950-4dd4-af7c-4333701ffb24-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.812204 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.812001 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e7a9993d-e950-4dd4-af7c-4333701ffb24-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.812204 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.812039 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e7a9993d-e950-4dd4-af7c-4333701ffb24-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.812204 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.812075 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e7a9993d-e950-4dd4-af7c-4333701ffb24-config-out\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.812204 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.812101 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e7a9993d-e950-4dd4-af7c-4333701ffb24-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.812204 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.812142 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7a9993d-e950-4dd4-af7c-4333701ffb24-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.812204 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.812184 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e7a9993d-e950-4dd4-af7c-4333701ffb24-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.812534 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.812208 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e7a9993d-e950-4dd4-af7c-4333701ffb24-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.812534 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.812232 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e7a9993d-e950-4dd4-af7c-4333701ffb24-web-config\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.812534 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.812252 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wml7\" (UniqueName: \"kubernetes.io/projected/e7a9993d-e950-4dd4-af7c-4333701ffb24-kube-api-access-2wml7\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.812534 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.812276 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e7a9993d-e950-4dd4-af7c-4333701ffb24-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.812534 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.812299 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e7a9993d-e950-4dd4-af7c-4333701ffb24-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.812534 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.812324 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e7a9993d-e950-4dd4-af7c-4333701ffb24-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.812534 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.812354 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7a9993d-e950-4dd4-af7c-4333701ffb24-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.812534 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.812378 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e7a9993d-e950-4dd4-af7c-4333701ffb24-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.812534 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.812410 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e7a9993d-e950-4dd4-af7c-4333701ffb24-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.812534 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.812460 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e7a9993d-e950-4dd4-af7c-4333701ffb24-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.812534 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.812489 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7a9993d-e950-4dd4-af7c-4333701ffb24-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.812534 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.812513 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7a9993d-e950-4dd4-af7c-4333701ffb24-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.813150 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.812931 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7a9993d-e950-4dd4-af7c-4333701ffb24-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.813977 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.813451 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7a9993d-e950-4dd4-af7c-4333701ffb24-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.813977 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.813473 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e7a9993d-e950-4dd4-af7c-4333701ffb24-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.814394 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.814364 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e7a9993d-e950-4dd4-af7c-4333701ffb24-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.815729 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.814769 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7a9993d-e950-4dd4-af7c-4333701ffb24-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.815729 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.814891 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e7a9993d-e950-4dd4-af7c-4333701ffb24-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.815729 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.815682 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7a9993d-e950-4dd4-af7c-4333701ffb24-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.816071 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.816044 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e7a9993d-e950-4dd4-af7c-4333701ffb24-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.816507 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.816480 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e7a9993d-e950-4dd4-af7c-4333701ffb24-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.817597 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.817565 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e7a9993d-e950-4dd4-af7c-4333701ffb24-config\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.817719 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.817565 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e7a9993d-e950-4dd4-af7c-4333701ffb24-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.817801 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.817775 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e7a9993d-e950-4dd4-af7c-4333701ffb24-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.818039 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.818015 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e7a9993d-e950-4dd4-af7c-4333701ffb24-config-out\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.818127 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.818019 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e7a9993d-e950-4dd4-af7c-4333701ffb24-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.818188 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.818175 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e7a9993d-e950-4dd4-af7c-4333701ffb24-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.818445 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.818427 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e7a9993d-e950-4dd4-af7c-4333701ffb24-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.818569 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.818542 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e7a9993d-e950-4dd4-af7c-4333701ffb24-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.818708 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.818600 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e7a9993d-e950-4dd4-af7c-4333701ffb24-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.819094 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.819075 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e7a9993d-e950-4dd4-af7c-4333701ffb24-web-config\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:26.855582 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.855552 5953 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e" path="/var/lib/kubelet/pods/d9c2dd4c-0b6e-4560-9b46-c62ee49cf49e/volumes" Nov 04 03:18:26.880236 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:26.880208 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wml7\" (UniqueName: \"kubernetes.io/projected/e7a9993d-e950-4dd4-af7c-4333701ffb24-kube-api-access-2wml7\") pod \"prometheus-k8s-0\" (UID: \"e7a9993d-e950-4dd4-af7c-4333701ffb24\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:27.154738 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:27.154706 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:27.587655 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:27.587599 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 03:18:27.591074 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:18:27.591049 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7a9993d_e950_4dd4_af7c_4333701ffb24.slice/crio-e00f527427ef707a67c581a95a40154dcf5cd2ea76fb8298be1c6d4392529be5 WatchSource:0}: Error finding container e00f527427ef707a67c581a95a40154dcf5cd2ea76fb8298be1c6d4392529be5: Status 404 returned error can't find the container with id e00f527427ef707a67c581a95a40154dcf5cd2ea76fb8298be1c6d4392529be5 Nov 04 03:18:28.071994 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:28.071956 5953 generic.go:356] "Generic (PLEG): container finished" podID="e7a9993d-e950-4dd4-af7c-4333701ffb24" containerID="ac65da43c659268928e275623f96def2beaad820ff4a20369e6dc84df2824b6e" exitCode=0 Nov 04 03:18:28.072173 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:28.072033 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e7a9993d-e950-4dd4-af7c-4333701ffb24","Type":"ContainerDied","Data":"ac65da43c659268928e275623f96def2beaad820ff4a20369e6dc84df2824b6e"} Nov 04 03:18:28.072173 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:28.072105 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e7a9993d-e950-4dd4-af7c-4333701ffb24","Type":"ContainerStarted","Data":"e00f527427ef707a67c581a95a40154dcf5cd2ea76fb8298be1c6d4392529be5"} Nov 04 03:18:29.080070 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:29.080037 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e7a9993d-e950-4dd4-af7c-4333701ffb24","Type":"ContainerStarted","Data":"a20e4392e07667c9b77020ada101c6c7a1bce3b3f9a5c00484a6a310357b2404"} Nov 04 03:18:29.080475 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:29.080104 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e7a9993d-e950-4dd4-af7c-4333701ffb24","Type":"ContainerStarted","Data":"0c2af92e66fe805ae7ecaf5f318a7fdeaf60f436526edcf29737c846096713d6"} Nov 04 03:18:29.080475 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:29.080132 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e7a9993d-e950-4dd4-af7c-4333701ffb24","Type":"ContainerStarted","Data":"5916e6d5d38312ead720d1c4ef17fa9bfa05c398a6ad01b5556f0a110449543f"} Nov 04 03:18:29.080475 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:29.080156 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e7a9993d-e950-4dd4-af7c-4333701ffb24","Type":"ContainerStarted","Data":"1719d1e52170079ea731cd25fbca511099a3f239917a02ba875d0c161cbeff41"} Nov 04 03:18:29.080475 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:29.080180 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e7a9993d-e950-4dd4-af7c-4333701ffb24","Type":"ContainerStarted","Data":"00226923a1eb224bc9ca37bf53e11bfd4db50d2efffe9bf710a596fe992800ad"} Nov 04 03:18:29.080475 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:29.080203 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e7a9993d-e950-4dd4-af7c-4333701ffb24","Type":"ContainerStarted","Data":"90b07fa7875b1d2a3173cbbb58808df2440fbbd54681ae5aed62b8bf6cb2d3f4"} Nov 04 03:18:29.213673 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:29.213599 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.195974166 podStartE2EDuration="3.213586193s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.612027ms" isStatefulPod=false podCreationTimestamp="2026-11-04 03:18:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:18:29.213218174 +0000 UTC m=+34.487848937" watchObservedRunningTime="2026-11-04 03:18:29.213586193 +0000 UTC m=+34.488216954" Nov 04 03:18:30.079874 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:30.079845 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2hjpn" Nov 04 03:18:32.154829 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:32.154785 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:46.383942 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:18:46.383901 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:18:56.330417 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:56.330359 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:56.346975 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:56.346949 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:18:57.171873 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:18:57.171841 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 03:19:16.383193 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:19:16.383158 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:19:46.386194 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:19:46.386161 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:20:16.384613 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:20:16.384573 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:20:46.384814 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:20:46.384778 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:21:16.384584 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:21:16.384548 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:21:30.671566 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:30.671528 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-45lrf"] Nov 04 03:21:30.673828 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:30.673808 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-45lrf" Nov 04 03:21:30.723726 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:30.723690 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkjrp\" (UniqueName: \"kubernetes.io/projected/9cdb0479-a378-4440-9597-74d47926c8a0-kube-api-access-bkjrp\") pod \"opendatahub-operator-controller-manager-549fcdc555-45lrf\" (UID: \"9cdb0479-a378-4440-9597-74d47926c8a0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-45lrf" Nov 04 03:21:30.723920 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:30.723740 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9cdb0479-a378-4440-9597-74d47926c8a0-tmp\") pod \"opendatahub-operator-controller-manager-549fcdc555-45lrf\" (UID: \"9cdb0479-a378-4440-9597-74d47926c8a0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-45lrf" Nov 04 03:21:30.723920 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:30.723811 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9cdb0479-a378-4440-9597-74d47926c8a0-manifests\") pod \"opendatahub-operator-controller-manager-549fcdc555-45lrf\" (UID: \"9cdb0479-a378-4440-9597-74d47926c8a0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-45lrf" Nov 04 03:21:30.723920 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:30.723839 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9cdb0479-a378-4440-9597-74d47926c8a0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-549fcdc555-45lrf\" (UID: \"9cdb0479-a378-4440-9597-74d47926c8a0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-45lrf" Nov 04 03:21:30.723920 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:30.723857 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9cdb0479-a378-4440-9597-74d47926c8a0-webhook-cert\") pod \"opendatahub-operator-controller-manager-549fcdc555-45lrf\" (UID: \"9cdb0479-a378-4440-9597-74d47926c8a0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-45lrf" Nov 04 03:21:30.723920 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:30.723871 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9cdb0479-a378-4440-9597-74d47926c8a0-metrics-cert\") pod \"opendatahub-operator-controller-manager-549fcdc555-45lrf\" (UID: \"9cdb0479-a378-4440-9597-74d47926c8a0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-45lrf" Nov 04 03:21:30.727583 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:30.727564 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 04 03:21:30.727666 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:30.727585 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 04 03:21:30.728058 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:30.728043 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-fqc69\"" Nov 04 03:21:30.777948 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:30.777926 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 04 03:21:30.779271 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:30.779253 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-45lrf"] Nov 04 03:21:30.794969 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:30.794951 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 04 03:21:30.795029 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:30.794996 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 04 03:21:30.825014 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:30.824985 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkjrp\" (UniqueName: \"kubernetes.io/projected/9cdb0479-a378-4440-9597-74d47926c8a0-kube-api-access-bkjrp\") pod \"opendatahub-operator-controller-manager-549fcdc555-45lrf\" (UID: \"9cdb0479-a378-4440-9597-74d47926c8a0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-45lrf" Nov 04 03:21:30.825133 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:30.825028 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9cdb0479-a378-4440-9597-74d47926c8a0-tmp\") pod \"opendatahub-operator-controller-manager-549fcdc555-45lrf\" (UID: \"9cdb0479-a378-4440-9597-74d47926c8a0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-45lrf" Nov 04 03:21:30.825133 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:30.825063 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9cdb0479-a378-4440-9597-74d47926c8a0-manifests\") pod \"opendatahub-operator-controller-manager-549fcdc555-45lrf\" (UID: \"9cdb0479-a378-4440-9597-74d47926c8a0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-45lrf" Nov 04 03:21:30.825133 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:30.825089 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9cdb0479-a378-4440-9597-74d47926c8a0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-549fcdc555-45lrf\" (UID: \"9cdb0479-a378-4440-9597-74d47926c8a0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-45lrf" Nov 04 03:21:30.825133 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:30.825108 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9cdb0479-a378-4440-9597-74d47926c8a0-webhook-cert\") pod \"opendatahub-operator-controller-manager-549fcdc555-45lrf\" (UID: \"9cdb0479-a378-4440-9597-74d47926c8a0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-45lrf" Nov 04 03:21:30.825330 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:30.825133 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9cdb0479-a378-4440-9597-74d47926c8a0-metrics-cert\") pod \"opendatahub-operator-controller-manager-549fcdc555-45lrf\" (UID: \"9cdb0479-a378-4440-9597-74d47926c8a0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-45lrf" Nov 04 03:21:30.825503 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:30.825476 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9cdb0479-a378-4440-9597-74d47926c8a0-manifests\") pod \"opendatahub-operator-controller-manager-549fcdc555-45lrf\" (UID: \"9cdb0479-a378-4440-9597-74d47926c8a0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-45lrf" Nov 04 03:21:30.827492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:30.827469 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9cdb0479-a378-4440-9597-74d47926c8a0-webhook-cert\") pod \"opendatahub-operator-controller-manager-549fcdc555-45lrf\" (UID: \"9cdb0479-a378-4440-9597-74d47926c8a0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-45lrf" Nov 04 03:21:30.827667 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:30.827644 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9cdb0479-a378-4440-9597-74d47926c8a0-metrics-cert\") pod \"opendatahub-operator-controller-manager-549fcdc555-45lrf\" (UID: \"9cdb0479-a378-4440-9597-74d47926c8a0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-45lrf" Nov 04 03:21:30.827750 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:30.827741 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9cdb0479-a378-4440-9597-74d47926c8a0-tmp\") pod \"opendatahub-operator-controller-manager-549fcdc555-45lrf\" (UID: \"9cdb0479-a378-4440-9597-74d47926c8a0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-45lrf" Nov 04 03:21:30.827983 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:30.827964 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9cdb0479-a378-4440-9597-74d47926c8a0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-549fcdc555-45lrf\" (UID: \"9cdb0479-a378-4440-9597-74d47926c8a0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-45lrf" Nov 04 03:21:30.887751 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:30.887715 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkjrp\" (UniqueName: \"kubernetes.io/projected/9cdb0479-a378-4440-9597-74d47926c8a0-kube-api-access-bkjrp\") pod \"opendatahub-operator-controller-manager-549fcdc555-45lrf\" (UID: \"9cdb0479-a378-4440-9597-74d47926c8a0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-45lrf" Nov 04 03:21:30.986160 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:30.986072 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-45lrf" Nov 04 03:21:31.479854 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:31.479820 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-45lrf"] Nov 04 03:21:31.487033 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:21:31.487009 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cdb0479_a378_4440_9597_74d47926c8a0.slice/crio-2bd38fdeae630adb5048745b1dec6e6f6c3c359b8bd3d9c0470e0883e8ae3f8d WatchSource:0}: Error finding container 2bd38fdeae630adb5048745b1dec6e6f6c3c359b8bd3d9c0470e0883e8ae3f8d: Status 404 returned error can't find the container with id 2bd38fdeae630adb5048745b1dec6e6f6c3c359b8bd3d9c0470e0883e8ae3f8d Nov 04 03:21:31.488774 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:31.488756 5953 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 03:21:31.749345 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:31.749267 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-45lrf" event={"ID":"9cdb0479-a378-4440-9597-74d47926c8a0","Type":"ContainerStarted","Data":"2bd38fdeae630adb5048745b1dec6e6f6c3c359b8bd3d9c0470e0883e8ae3f8d"} Nov 04 03:21:35.767803 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:35.767768 5953 generic.go:356] "Generic (PLEG): container finished" podID="9cdb0479-a378-4440-9597-74d47926c8a0" containerID="d9beb705b0404683e39dd9c8160d29e9298b590edd77aa65ee9add86e99c3d2e" exitCode=0 Nov 04 03:21:35.768183 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:35.767823 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-45lrf" event={"ID":"9cdb0479-a378-4440-9597-74d47926c8a0","Type":"ContainerDied","Data":"d9beb705b0404683e39dd9c8160d29e9298b590edd77aa65ee9add86e99c3d2e"} Nov 04 03:21:36.161106 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:36.161013 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-s9qz2"] Nov 04 03:21:36.163176 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:36.163153 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-s9qz2" Nov 04 03:21:36.218441 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:36.218398 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-hr2w6\"" Nov 04 03:21:36.218643 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:36.218471 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 04 03:21:36.218643 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:36.218475 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 04 03:21:36.223225 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:36.223201 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-s9qz2"] Nov 04 03:21:36.275848 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:36.275815 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8q7b\" (UniqueName: \"kubernetes.io/projected/a69c0085-a9fe-4797-b444-f66d4cd613cd-kube-api-access-s8q7b\") pod \"jobset-operator-674cd59dc5-s9qz2\" (UID: \"a69c0085-a9fe-4797-b444-f66d4cd613cd\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-s9qz2" Nov 04 03:21:36.276036 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:36.275870 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a69c0085-a9fe-4797-b444-f66d4cd613cd-tmp\") pod \"jobset-operator-674cd59dc5-s9qz2\" (UID: \"a69c0085-a9fe-4797-b444-f66d4cd613cd\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-s9qz2" Nov 04 03:21:36.376776 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:36.376736 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8q7b\" (UniqueName: \"kubernetes.io/projected/a69c0085-a9fe-4797-b444-f66d4cd613cd-kube-api-access-s8q7b\") pod \"jobset-operator-674cd59dc5-s9qz2\" (UID: \"a69c0085-a9fe-4797-b444-f66d4cd613cd\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-s9qz2" Nov 04 03:21:36.376776 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:36.376783 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a69c0085-a9fe-4797-b444-f66d4cd613cd-tmp\") pod \"jobset-operator-674cd59dc5-s9qz2\" (UID: \"a69c0085-a9fe-4797-b444-f66d4cd613cd\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-s9qz2" Nov 04 03:21:36.377145 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:36.377128 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a69c0085-a9fe-4797-b444-f66d4cd613cd-tmp\") pod \"jobset-operator-674cd59dc5-s9qz2\" (UID: \"a69c0085-a9fe-4797-b444-f66d4cd613cd\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-s9qz2" Nov 04 03:21:36.438365 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:36.438295 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8q7b\" (UniqueName: \"kubernetes.io/projected/a69c0085-a9fe-4797-b444-f66d4cd613cd-kube-api-access-s8q7b\") pod \"jobset-operator-674cd59dc5-s9qz2\" (UID: \"a69c0085-a9fe-4797-b444-f66d4cd613cd\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-s9qz2" Nov 04 03:21:36.484467 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:36.484430 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-s9qz2" Nov 04 03:21:36.773337 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:36.773309 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-45lrf" event={"ID":"9cdb0479-a378-4440-9597-74d47926c8a0","Type":"ContainerStarted","Data":"80ea723e1a4dee3eac44445123732cfaee9fec508562e504cc9138d989af2297"} Nov 04 03:21:36.773712 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:36.773438 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-45lrf" Nov 04 03:21:36.859548 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:36.859515 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-s9qz2"] Nov 04 03:21:36.862808 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:21:36.862781 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda69c0085_a9fe_4797_b444_f66d4cd613cd.slice/crio-2b6afcf079e8152aea7bb503a9576ea6d50fa7d8bb4bc3e19d9a08ecd0ade9c9 WatchSource:0}: Error finding container 2b6afcf079e8152aea7bb503a9576ea6d50fa7d8bb4bc3e19d9a08ecd0ade9c9: Status 404 returned error can't find the container with id 2b6afcf079e8152aea7bb503a9576ea6d50fa7d8bb4bc3e19d9a08ecd0ade9c9 Nov 04 03:21:36.928713 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:36.928660 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-45lrf" podStartSLOduration=2.213185137 podStartE2EDuration="6.928643333s" totalImagesPullingTime="4.233625584s" totalInitContainerRuntime="481.832612ms" isStatefulPod=false podCreationTimestamp="2026-11-04 03:21:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:21:36.926912486 +0000 UTC m=+222.201543248" watchObservedRunningTime="2026-11-04 03:21:36.928643333 +0000 UTC m=+222.203274095" Nov 04 03:21:37.779212 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:37.779178 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-s9qz2" event={"ID":"a69c0085-a9fe-4797-b444-f66d4cd613cd","Type":"ContainerStarted","Data":"2b6afcf079e8152aea7bb503a9576ea6d50fa7d8bb4bc3e19d9a08ecd0ade9c9"} Nov 04 03:21:40.795212 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:40.795168 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-s9qz2" event={"ID":"a69c0085-a9fe-4797-b444-f66d4cd613cd","Type":"ContainerStarted","Data":"d34227999027df149b1ed4990528353842430c235689db7ce8af27d304f5c5e2"} Nov 04 03:21:40.954701 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:40.954539 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-s9qz2" podStartSLOduration=1.151281805 podStartE2EDuration="4.954519018s" totalImagesPullingTime="3.803237213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:21:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:21:40.954167645 +0000 UTC m=+226.228798448" watchObservedRunningTime="2026-11-04 03:21:40.954519018 +0000 UTC m=+226.229149778" Nov 04 03:21:46.384157 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:21:46.384116 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:21:47.781778 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:21:47.781748 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-45lrf" Nov 04 03:22:00.200072 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:00.200037 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-vxhx8"] Nov 04 03:22:00.207559 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:00.207538 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vxhx8" Nov 04 03:22:00.264292 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:00.264263 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 04 03:22:00.302704 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:00.302667 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnq99\" (UniqueName: \"kubernetes.io/projected/b0a6e662-9a7e-4a6a-a4f6-a804a708f7ae-kube-api-access-rnq99\") pod \"cert-manager-cainjector-5c57bbd4fc-vxhx8\" (UID: \"b0a6e662-9a7e-4a6a-a4f6-a804a708f7ae\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vxhx8" Nov 04 03:22:00.302704 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:00.302711 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b0a6e662-9a7e-4a6a-a4f6-a804a708f7ae-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-vxhx8\" (UID: \"b0a6e662-9a7e-4a6a-a4f6-a804a708f7ae\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vxhx8" Nov 04 03:22:00.313997 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:00.313965 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 04 03:22:00.314100 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:00.314031 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-dsrq9\"" Nov 04 03:22:00.314813 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:00.314796 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-vxhx8"] Nov 04 03:22:00.404283 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:00.404254 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnq99\" (UniqueName: \"kubernetes.io/projected/b0a6e662-9a7e-4a6a-a4f6-a804a708f7ae-kube-api-access-rnq99\") pod \"cert-manager-cainjector-5c57bbd4fc-vxhx8\" (UID: \"b0a6e662-9a7e-4a6a-a4f6-a804a708f7ae\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vxhx8" Nov 04 03:22:00.404461 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:00.404300 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b0a6e662-9a7e-4a6a-a4f6-a804a708f7ae-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-vxhx8\" (UID: \"b0a6e662-9a7e-4a6a-a4f6-a804a708f7ae\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vxhx8" Nov 04 03:22:00.473903 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:00.473833 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b0a6e662-9a7e-4a6a-a4f6-a804a708f7ae-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-vxhx8\" (UID: \"b0a6e662-9a7e-4a6a-a4f6-a804a708f7ae\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vxhx8" Nov 04 03:22:00.475004 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:00.474970 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnq99\" (UniqueName: \"kubernetes.io/projected/b0a6e662-9a7e-4a6a-a4f6-a804a708f7ae-kube-api-access-rnq99\") pod \"cert-manager-cainjector-5c57bbd4fc-vxhx8\" (UID: \"b0a6e662-9a7e-4a6a-a4f6-a804a708f7ae\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vxhx8" Nov 04 03:22:00.518213 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:00.518175 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vxhx8" Nov 04 03:22:00.889567 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:00.889536 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-vxhx8"] Nov 04 03:22:00.892910 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:22:00.892883 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0a6e662_9a7e_4a6a_a4f6_a804a708f7ae.slice/crio-8f8e7017fd362bcae0cd323c1292091222a6ab2e1d8061801595259111baf48e WatchSource:0}: Error finding container 8f8e7017fd362bcae0cd323c1292091222a6ab2e1d8061801595259111baf48e: Status 404 returned error can't find the container with id 8f8e7017fd362bcae0cd323c1292091222a6ab2e1d8061801595259111baf48e Nov 04 03:22:01.279173 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:01.279135 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5d77786fb9-9gmhr"] Nov 04 03:22:01.282234 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:01.282214 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d77786fb9-9gmhr" Nov 04 03:22:01.311919 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:01.311891 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7f96f38-4540-4d54-a792-35996a8767c8-tls-certs\") pod \"kube-auth-proxy-5d77786fb9-9gmhr\" (UID: \"b7f96f38-4540-4d54-a792-35996a8767c8\") " pod="openshift-ingress/kube-auth-proxy-5d77786fb9-9gmhr" Nov 04 03:22:01.312095 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:01.311939 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7f96f38-4540-4d54-a792-35996a8767c8-tmp\") pod \"kube-auth-proxy-5d77786fb9-9gmhr\" (UID: \"b7f96f38-4540-4d54-a792-35996a8767c8\") " pod="openshift-ingress/kube-auth-proxy-5d77786fb9-9gmhr" Nov 04 03:22:01.312095 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:01.312078 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5skfx\" (UniqueName: \"kubernetes.io/projected/b7f96f38-4540-4d54-a792-35996a8767c8-kube-api-access-5skfx\") pod \"kube-auth-proxy-5d77786fb9-9gmhr\" (UID: \"b7f96f38-4540-4d54-a792-35996a8767c8\") " pod="openshift-ingress/kube-auth-proxy-5d77786fb9-9gmhr" Nov 04 03:22:01.338897 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:01.338860 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 04 03:22:01.339034 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:01.338970 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 04 03:22:01.344391 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:01.344357 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d77786fb9-9gmhr"] Nov 04 03:22:01.413328 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:01.413297 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7f96f38-4540-4d54-a792-35996a8767c8-tmp\") pod \"kube-auth-proxy-5d77786fb9-9gmhr\" (UID: \"b7f96f38-4540-4d54-a792-35996a8767c8\") " pod="openshift-ingress/kube-auth-proxy-5d77786fb9-9gmhr" Nov 04 03:22:01.413534 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:01.413364 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5skfx\" (UniqueName: \"kubernetes.io/projected/b7f96f38-4540-4d54-a792-35996a8767c8-kube-api-access-5skfx\") pod \"kube-auth-proxy-5d77786fb9-9gmhr\" (UID: \"b7f96f38-4540-4d54-a792-35996a8767c8\") " pod="openshift-ingress/kube-auth-proxy-5d77786fb9-9gmhr" Nov 04 03:22:01.413534 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:01.413402 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7f96f38-4540-4d54-a792-35996a8767c8-tls-certs\") pod \"kube-auth-proxy-5d77786fb9-9gmhr\" (UID: \"b7f96f38-4540-4d54-a792-35996a8767c8\") " pod="openshift-ingress/kube-auth-proxy-5d77786fb9-9gmhr" Nov 04 03:22:01.415790 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:01.415765 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7f96f38-4540-4d54-a792-35996a8767c8-tmp\") pod \"kube-auth-proxy-5d77786fb9-9gmhr\" (UID: \"b7f96f38-4540-4d54-a792-35996a8767c8\") " pod="openshift-ingress/kube-auth-proxy-5d77786fb9-9gmhr" Nov 04 03:22:01.422478 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:01.422461 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 04 03:22:01.422727 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:01.422711 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 04 03:22:01.422852 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:01.422820 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-trx6p\"" Nov 04 03:22:01.426053 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:01.426033 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7f96f38-4540-4d54-a792-35996a8767c8-tls-certs\") pod \"kube-auth-proxy-5d77786fb9-9gmhr\" (UID: \"b7f96f38-4540-4d54-a792-35996a8767c8\") " pod="openshift-ingress/kube-auth-proxy-5d77786fb9-9gmhr" Nov 04 03:22:01.476470 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:01.476435 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5skfx\" (UniqueName: \"kubernetes.io/projected/b7f96f38-4540-4d54-a792-35996a8767c8-kube-api-access-5skfx\") pod \"kube-auth-proxy-5d77786fb9-9gmhr\" (UID: \"b7f96f38-4540-4d54-a792-35996a8767c8\") " pod="openshift-ingress/kube-auth-proxy-5d77786fb9-9gmhr" Nov 04 03:22:01.594452 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:01.594364 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d77786fb9-9gmhr" Nov 04 03:22:01.882768 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:01.882656 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vxhx8" event={"ID":"b0a6e662-9a7e-4a6a-a4f6-a804a708f7ae","Type":"ContainerStarted","Data":"8f8e7017fd362bcae0cd323c1292091222a6ab2e1d8061801595259111baf48e"} Nov 04 03:22:01.973553 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:01.972247 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d77786fb9-9gmhr"] Nov 04 03:22:01.976693 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:22:01.976660 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7f96f38_4540_4d54_a792_35996a8767c8.slice/crio-67cab7a8904d0b98d13e86e92c161cc9837f0eda91ec2ee5c4ce87b49d02d561 WatchSource:0}: Error finding container 67cab7a8904d0b98d13e86e92c161cc9837f0eda91ec2ee5c4ce87b49d02d561: Status 404 returned error can't find the container with id 67cab7a8904d0b98d13e86e92c161cc9837f0eda91ec2ee5c4ce87b49d02d561 Nov 04 03:22:02.310870 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:02.310842 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-dcl2t"] Nov 04 03:22:02.313566 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:02.313549 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dcl2t" Nov 04 03:22:02.372678 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:02.372644 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 04 03:22:02.372678 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:02.372675 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 04 03:22:02.373005 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:02.372986 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-64g6p\"" Nov 04 03:22:02.376948 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:02.376914 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-dcl2t"] Nov 04 03:22:02.421749 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:02.421711 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw7mf\" (UniqueName: \"kubernetes.io/projected/0c122db7-3d8a-4aec-b046-8e02811c7082-kube-api-access-sw7mf\") pod \"limitador-operator-controller-manager-5778bb4449-dcl2t\" (UID: \"0c122db7-3d8a-4aec-b046-8e02811c7082\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dcl2t" Nov 04 03:22:02.522801 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:02.522767 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw7mf\" (UniqueName: \"kubernetes.io/projected/0c122db7-3d8a-4aec-b046-8e02811c7082-kube-api-access-sw7mf\") pod \"limitador-operator-controller-manager-5778bb4449-dcl2t\" (UID: \"0c122db7-3d8a-4aec-b046-8e02811c7082\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dcl2t" Nov 04 03:22:02.584919 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:02.584845 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw7mf\" (UniqueName: \"kubernetes.io/projected/0c122db7-3d8a-4aec-b046-8e02811c7082-kube-api-access-sw7mf\") pod \"limitador-operator-controller-manager-5778bb4449-dcl2t\" (UID: \"0c122db7-3d8a-4aec-b046-8e02811c7082\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dcl2t" Nov 04 03:22:02.626021 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:02.625978 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dcl2t" Nov 04 03:22:02.892847 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:02.892718 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d77786fb9-9gmhr" event={"ID":"b7f96f38-4540-4d54-a792-35996a8767c8","Type":"ContainerStarted","Data":"67cab7a8904d0b98d13e86e92c161cc9837f0eda91ec2ee5c4ce87b49d02d561"} Nov 04 03:22:03.023988 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:03.023949 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-dcl2t"] Nov 04 03:22:03.027792 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:22:03.027760 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c122db7_3d8a_4aec_b046_8e02811c7082.slice/crio-528cfc80b13cb55716b61b4bda7fcfc3eb4e81f3484ee448229be2fe0f7a740a WatchSource:0}: Error finding container 528cfc80b13cb55716b61b4bda7fcfc3eb4e81f3484ee448229be2fe0f7a740a: Status 404 returned error can't find the container with id 528cfc80b13cb55716b61b4bda7fcfc3eb4e81f3484ee448229be2fe0f7a740a Nov 04 03:22:03.902825 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:03.902733 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dcl2t" event={"ID":"0c122db7-3d8a-4aec-b046-8e02811c7082","Type":"ContainerStarted","Data":"528cfc80b13cb55716b61b4bda7fcfc3eb4e81f3484ee448229be2fe0f7a740a"} Nov 04 03:22:05.703591 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:05.703556 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-xzvqn"] Nov 04 03:22:05.706255 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:05.706225 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-xzvqn" Nov 04 03:22:05.755512 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:05.755472 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glkz8\" (UniqueName: \"kubernetes.io/projected/eef0a147-2a4f-48fe-bc09-c96b8d05e888-kube-api-access-glkz8\") pod \"authorino-operator-7b7bb4bc77-xzvqn\" (UID: \"eef0a147-2a4f-48fe-bc09-c96b8d05e888\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-xzvqn" Nov 04 03:22:05.761943 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:05.761906 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-lcs5r\"" Nov 04 03:22:05.813338 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:05.813310 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-xzvqn"] Nov 04 03:22:05.856516 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:05.856476 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glkz8\" (UniqueName: \"kubernetes.io/projected/eef0a147-2a4f-48fe-bc09-c96b8d05e888-kube-api-access-glkz8\") pod \"authorino-operator-7b7bb4bc77-xzvqn\" (UID: \"eef0a147-2a4f-48fe-bc09-c96b8d05e888\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-xzvqn" Nov 04 03:22:05.916596 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:05.916561 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d77786fb9-9gmhr" event={"ID":"b7f96f38-4540-4d54-a792-35996a8767c8","Type":"ContainerStarted","Data":"bd3e4e60b0de3ace71394b8d244b3919356d63e112dd5c259fd038f9e3066475"} Nov 04 03:22:05.923772 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:05.923739 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glkz8\" (UniqueName: \"kubernetes.io/projected/eef0a147-2a4f-48fe-bc09-c96b8d05e888-kube-api-access-glkz8\") pod \"authorino-operator-7b7bb4bc77-xzvqn\" (UID: \"eef0a147-2a4f-48fe-bc09-c96b8d05e888\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-xzvqn" Nov 04 03:22:06.021119 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:06.021086 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-xzvqn" Nov 04 03:22:06.074665 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:06.070961 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5d77786fb9-9gmhr" podStartSLOduration=1.311075549 podStartE2EDuration="5.070946902s" totalImagesPullingTime="3.759871353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:22:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:22:06.069264864 +0000 UTC m=+251.343895617" watchObservedRunningTime="2026-11-04 03:22:06.070946902 +0000 UTC m=+251.345577658" Nov 04 03:22:06.473702 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:06.473665 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-xzvqn"] Nov 04 03:22:06.477285 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:22:06.477258 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef0a147_2a4f_48fe_bc09_c96b8d05e888.slice/crio-8f081a5817e9e3f2ac5f23a070c6fea10ce40a364d80ebd813ec56e2765e268f WatchSource:0}: Error finding container 8f081a5817e9e3f2ac5f23a070c6fea10ce40a364d80ebd813ec56e2765e268f: Status 404 returned error can't find the container with id 8f081a5817e9e3f2ac5f23a070c6fea10ce40a364d80ebd813ec56e2765e268f Nov 04 03:22:06.925402 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:06.925300 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-xzvqn" event={"ID":"eef0a147-2a4f-48fe-bc09-c96b8d05e888","Type":"ContainerStarted","Data":"8f081a5817e9e3f2ac5f23a070c6fea10ce40a364d80ebd813ec56e2765e268f"} Nov 04 03:22:06.928148 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:06.928120 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vxhx8" event={"ID":"b0a6e662-9a7e-4a6a-a4f6-a804a708f7ae","Type":"ContainerStarted","Data":"ba57030622ba9f1cae60f5d0564e673df5bb9878cb9f68ffab335b19af840e43"} Nov 04 03:22:07.049315 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:07.049244 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vxhx8" podStartSLOduration=1.6628708159999999 podStartE2EDuration="7.049227865s" totalImagesPullingTime="5.386357049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:22:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:22:07.048264621 +0000 UTC m=+252.322895389" watchObservedRunningTime="2026-11-04 03:22:07.049227865 +0000 UTC m=+252.323858624" Nov 04 03:22:07.936327 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:07.936283 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dcl2t" event={"ID":"0c122db7-3d8a-4aec-b046-8e02811c7082","Type":"ContainerStarted","Data":"3654ad754773cf5d0849233b3bed07f6fb33cb89d804e4feeecd82d2f7daf7a5"} Nov 04 03:22:07.936853 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:07.936492 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dcl2t" Nov 04 03:22:08.095249 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:08.095179 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dcl2t" podStartSLOduration=2.245245034 podStartE2EDuration="6.095162244s" totalImagesPullingTime="3.84991721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:22:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:22:08.094236309 +0000 UTC m=+253.368867085" watchObservedRunningTime="2026-11-04 03:22:08.095162244 +0000 UTC m=+253.369793007" Nov 04 03:22:09.947428 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:09.947373 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-xzvqn" event={"ID":"eef0a147-2a4f-48fe-bc09-c96b8d05e888","Type":"ContainerStarted","Data":"a36e5ba8716ab9800ba4d034adb6352e7605d57aa2e1a497c8233749a25d4ccc"} Nov 04 03:22:09.947924 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:09.947525 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-xzvqn" Nov 04 03:22:10.065262 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:10.065199 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-xzvqn" podStartSLOduration=2.4474071410000002 podStartE2EDuration="5.065178306s" totalImagesPullingTime="2.617771165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:22:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:22:10.064095431 +0000 UTC m=+255.338726195" watchObservedRunningTime="2026-11-04 03:22:10.065178306 +0000 UTC m=+255.339809069" Nov 04 03:22:16.384606 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:22:16.384570 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:22:18.944092 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:18.944064 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-dcl2t" Nov 04 03:22:20.955955 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:20.955918 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-xzvqn" Nov 04 03:22:28.687864 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:28.687831 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-f6dp2"] Nov 04 03:22:28.803117 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:28.803080 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-f6dp2" Nov 04 03:22:28.804337 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:28.804309 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-f6dp2"] Nov 04 03:22:28.856572 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:28.856521 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-2r2dg\"" Nov 04 03:22:28.856790 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:28.856775 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 04 03:22:28.908853 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:28.908811 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 04 03:22:28.986138 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:28.986038 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2bc38436-c34c-4561-9463-07b5edc984e7-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-f6dp2\" (UID: \"2bc38436-c34c-4561-9463-07b5edc984e7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-f6dp2" Nov 04 03:22:28.986489 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:28.986466 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bc38436-c34c-4561-9463-07b5edc984e7-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-f6dp2\" (UID: \"2bc38436-c34c-4561-9463-07b5edc984e7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-f6dp2" Nov 04 03:22:28.986774 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:28.986754 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w67r8\" (UniqueName: \"kubernetes.io/projected/2bc38436-c34c-4561-9463-07b5edc984e7-kube-api-access-w67r8\") pod \"kserve-module-controller-manager-7566cf5fc8-f6dp2\" (UID: \"2bc38436-c34c-4561-9463-07b5edc984e7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-f6dp2" Nov 04 03:22:28.986978 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:28.986947 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2bc38436-c34c-4561-9463-07b5edc984e7-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-f6dp2\" (UID: \"2bc38436-c34c-4561-9463-07b5edc984e7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-f6dp2" Nov 04 03:22:29.010110 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:29.010074 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 04 03:22:29.088310 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:29.088277 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w67r8\" (UniqueName: \"kubernetes.io/projected/2bc38436-c34c-4561-9463-07b5edc984e7-kube-api-access-w67r8\") pod \"kserve-module-controller-manager-7566cf5fc8-f6dp2\" (UID: \"2bc38436-c34c-4561-9463-07b5edc984e7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-f6dp2" Nov 04 03:22:29.088310 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:29.088316 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2bc38436-c34c-4561-9463-07b5edc984e7-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-f6dp2\" (UID: \"2bc38436-c34c-4561-9463-07b5edc984e7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-f6dp2" Nov 04 03:22:29.088545 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:29.088349 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2bc38436-c34c-4561-9463-07b5edc984e7-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-f6dp2\" (UID: \"2bc38436-c34c-4561-9463-07b5edc984e7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-f6dp2" Nov 04 03:22:29.088545 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:29.088383 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bc38436-c34c-4561-9463-07b5edc984e7-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-f6dp2\" (UID: \"2bc38436-c34c-4561-9463-07b5edc984e7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-f6dp2" Nov 04 03:22:29.088771 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:29.088753 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bc38436-c34c-4561-9463-07b5edc984e7-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-f6dp2\" (UID: \"2bc38436-c34c-4561-9463-07b5edc984e7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-f6dp2" Nov 04 03:22:29.088855 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:29.088834 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2bc38436-c34c-4561-9463-07b5edc984e7-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-f6dp2\" (UID: \"2bc38436-c34c-4561-9463-07b5edc984e7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-f6dp2" Nov 04 03:22:29.089253 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:29.089234 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2bc38436-c34c-4561-9463-07b5edc984e7-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-f6dp2\" (UID: \"2bc38436-c34c-4561-9463-07b5edc984e7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-f6dp2" Nov 04 03:22:29.153083 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:29.153040 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w67r8\" (UniqueName: \"kubernetes.io/projected/2bc38436-c34c-4561-9463-07b5edc984e7-kube-api-access-w67r8\") pod \"kserve-module-controller-manager-7566cf5fc8-f6dp2\" (UID: \"2bc38436-c34c-4561-9463-07b5edc984e7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-f6dp2" Nov 04 03:22:29.434007 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:29.433971 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-f6dp2" Nov 04 03:22:29.875039 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:29.875002 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-f6dp2"] Nov 04 03:22:29.877927 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:22:29.877892 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bc38436_c34c_4561_9463_07b5edc984e7.slice/crio-031cacdfbb26c2d23be18937c4bc8613f3a14f4a63cdae917291b6a30309ba75 WatchSource:0}: Error finding container 031cacdfbb26c2d23be18937c4bc8613f3a14f4a63cdae917291b6a30309ba75: Status 404 returned error can't find the container with id 031cacdfbb26c2d23be18937c4bc8613f3a14f4a63cdae917291b6a30309ba75 Nov 04 03:22:30.043708 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:30.043674 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-f6dp2" event={"ID":"2bc38436-c34c-4561-9463-07b5edc984e7","Type":"ContainerStarted","Data":"031cacdfbb26c2d23be18937c4bc8613f3a14f4a63cdae917291b6a30309ba75"} Nov 04 03:22:32.054730 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:32.054692 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-f6dp2" event={"ID":"2bc38436-c34c-4561-9463-07b5edc984e7","Type":"ContainerStarted","Data":"245014392eeff9c31ac1d6f7c77a19f10277b4c1d12b402ca0931e9b0ffdd37c"} Nov 04 03:22:32.055181 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:32.054818 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-f6dp2" Nov 04 03:22:32.225577 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:32.225529 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-f6dp2" podStartSLOduration=2.374107342 podStartE2EDuration="4.225513767s" totalImagesPullingTime="1.851406425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:22:32.224485729 +0000 UTC m=+277.499116483" watchObservedRunningTime="2026-11-04 03:22:32.225513767 +0000 UTC m=+277.500144529" Nov 04 03:22:46.385100 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:22:46.385057 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:22:53.060923 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:53.060894 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-f6dp2" Nov 04 03:22:53.387042 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:53.386957 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-4ldrz"] Nov 04 03:22:53.389353 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:53.389330 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-4ldrz" Nov 04 03:22:53.442345 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:53.442305 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9ktvr\"" Nov 04 03:22:53.446527 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:53.446501 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-4ldrz"] Nov 04 03:22:53.508061 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:53.508023 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzdrs\" (UniqueName: \"kubernetes.io/projected/bd5cdc80-d609-4312-89c7-226da9334e2f-kube-api-access-rzdrs\") pod \"authorino-56c6c9ddd5-4ldrz\" (UID: \"bd5cdc80-d609-4312-89c7-226da9334e2f\") " pod="kuadrant-system/authorino-56c6c9ddd5-4ldrz" Nov 04 03:22:53.609441 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:53.609402 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzdrs\" (UniqueName: \"kubernetes.io/projected/bd5cdc80-d609-4312-89c7-226da9334e2f-kube-api-access-rzdrs\") pod \"authorino-56c6c9ddd5-4ldrz\" (UID: \"bd5cdc80-d609-4312-89c7-226da9334e2f\") " pod="kuadrant-system/authorino-56c6c9ddd5-4ldrz" Nov 04 03:22:53.671379 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:53.671312 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzdrs\" (UniqueName: \"kubernetes.io/projected/bd5cdc80-d609-4312-89c7-226da9334e2f-kube-api-access-rzdrs\") pod \"authorino-56c6c9ddd5-4ldrz\" (UID: \"bd5cdc80-d609-4312-89c7-226da9334e2f\") " pod="kuadrant-system/authorino-56c6c9ddd5-4ldrz" Nov 04 03:22:53.699757 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:53.699727 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-4ldrz" Nov 04 03:22:54.067830 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:54.067782 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-4ldrz"] Nov 04 03:22:54.070697 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:22:54.070659 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd5cdc80_d609_4312_89c7_226da9334e2f.slice/crio-26852e2ea59663c357e1212605dd75a4c4ce4004639ea58850475a1f931be486 WatchSource:0}: Error finding container 26852e2ea59663c357e1212605dd75a4c4ce4004639ea58850475a1f931be486: Status 404 returned error can't find the container with id 26852e2ea59663c357e1212605dd75a4c4ce4004639ea58850475a1f931be486 Nov 04 03:22:54.154826 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:54.154792 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-4ldrz" event={"ID":"bd5cdc80-d609-4312-89c7-226da9334e2f","Type":"ContainerStarted","Data":"26852e2ea59663c357e1212605dd75a4c4ce4004639ea58850475a1f931be486"} Nov 04 03:22:57.171811 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:57.171766 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-4ldrz" event={"ID":"bd5cdc80-d609-4312-89c7-226da9334e2f","Type":"ContainerStarted","Data":"f5b2b82891422224c6d9d7183b2345c4094d2e7d03e138955f7b99bbfee3b7b6"} Nov 04 03:22:57.287969 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:22:57.287916 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-4ldrz" podStartSLOduration=1.988328277 podStartE2EDuration="4.287901925s" totalImagesPullingTime="2.299573648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:22:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:22:57.287183252 +0000 UTC m=+302.561814013" watchObservedRunningTime="2026-11-04 03:22:57.287901925 +0000 UTC m=+302.562532684" Nov 04 03:23:16.384700 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:23:16.384658 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:23:35.979383 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:35.979333 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-w5ktz"] Nov 04 03:23:35.982921 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:35.982886 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-w5ktz" Nov 04 03:23:36.042014 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:36.041978 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-w5ktz"] Nov 04 03:23:36.086250 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:36.086213 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 04 03:23:36.086250 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:36.086221 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-zzdvm\"" Nov 04 03:23:36.093625 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:36.093596 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94939ede-2ec5-4439-b25b-30e7cf09a65e-tls-certs\") pod \"model-serving-api-75bd89fdfc-w5ktz\" (UID: \"94939ede-2ec5-4439-b25b-30e7cf09a65e\") " pod="opendatahub/model-serving-api-75bd89fdfc-w5ktz" Nov 04 03:23:36.093734 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:36.093661 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75h99\" (UniqueName: \"kubernetes.io/projected/94939ede-2ec5-4439-b25b-30e7cf09a65e-kube-api-access-75h99\") pod \"model-serving-api-75bd89fdfc-w5ktz\" (UID: \"94939ede-2ec5-4439-b25b-30e7cf09a65e\") " pod="opendatahub/model-serving-api-75bd89fdfc-w5ktz" Nov 04 03:23:36.195185 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:36.195149 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94939ede-2ec5-4439-b25b-30e7cf09a65e-tls-certs\") pod \"model-serving-api-75bd89fdfc-w5ktz\" (UID: \"94939ede-2ec5-4439-b25b-30e7cf09a65e\") " pod="opendatahub/model-serving-api-75bd89fdfc-w5ktz" Nov 04 03:23:36.195368 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:36.195194 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75h99\" (UniqueName: \"kubernetes.io/projected/94939ede-2ec5-4439-b25b-30e7cf09a65e-kube-api-access-75h99\") pod \"model-serving-api-75bd89fdfc-w5ktz\" (UID: \"94939ede-2ec5-4439-b25b-30e7cf09a65e\") " pod="opendatahub/model-serving-api-75bd89fdfc-w5ktz" Nov 04 03:23:36.197843 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:36.197817 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94939ede-2ec5-4439-b25b-30e7cf09a65e-tls-certs\") pod \"model-serving-api-75bd89fdfc-w5ktz\" (UID: \"94939ede-2ec5-4439-b25b-30e7cf09a65e\") " pod="opendatahub/model-serving-api-75bd89fdfc-w5ktz" Nov 04 03:23:36.256845 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:36.256813 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75h99\" (UniqueName: \"kubernetes.io/projected/94939ede-2ec5-4439-b25b-30e7cf09a65e-kube-api-access-75h99\") pod \"model-serving-api-75bd89fdfc-w5ktz\" (UID: \"94939ede-2ec5-4439-b25b-30e7cf09a65e\") " pod="opendatahub/model-serving-api-75bd89fdfc-w5ktz" Nov 04 03:23:36.295999 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:36.295962 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-w5ktz" Nov 04 03:23:36.663723 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:36.663611 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-w5ktz"] Nov 04 03:23:36.666563 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:23:36.666536 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94939ede_2ec5_4439_b25b_30e7cf09a65e.slice/crio-6be0b255ee52705a561207d38f45c73ea3c5642da504b3a6521d965f1ae91b9c WatchSource:0}: Error finding container 6be0b255ee52705a561207d38f45c73ea3c5642da504b3a6521d965f1ae91b9c: Status 404 returned error can't find the container with id 6be0b255ee52705a561207d38f45c73ea3c5642da504b3a6521d965f1ae91b9c Nov 04 03:23:37.362785 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:37.362747 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-w5ktz" event={"ID":"94939ede-2ec5-4439-b25b-30e7cf09a65e","Type":"ContainerStarted","Data":"6be0b255ee52705a561207d38f45c73ea3c5642da504b3a6521d965f1ae91b9c"} Nov 04 03:23:39.372839 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:39.372801 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-w5ktz" event={"ID":"94939ede-2ec5-4439-b25b-30e7cf09a65e","Type":"ContainerStarted","Data":"7b9a5214a186ae52f427aac2f85cc5328988e95e82456e18c731a154bca7cb01"} Nov 04 03:23:39.373248 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:39.372912 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-w5ktz" Nov 04 03:23:39.491205 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:39.491136 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-w5ktz" podStartSLOduration=2.170429635 podStartE2EDuration="4.491118222s" totalImagesPullingTime="2.320688587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:23:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:23:39.48980218 +0000 UTC m=+344.764432941" watchObservedRunningTime="2026-11-04 03:23:39.491118222 +0000 UTC m=+344.765748984" Nov 04 03:23:46.385018 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:23:46.384977 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:23:50.381579 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:50.381549 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-w5ktz" Nov 04 03:23:50.577242 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:50.577210 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-868857dbd9-6b89z"] Nov 04 03:23:50.580990 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:50.580970 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-868857dbd9-6b89z" Nov 04 03:23:50.634100 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:50.633984 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 04 03:23:50.634251 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:50.634165 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-dfrbx\"" Nov 04 03:23:50.634251 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:50.634171 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 04 03:23:50.634251 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:50.634171 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 04 03:23:50.640305 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:50.640274 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-868857dbd9-6b89z"] Nov 04 03:23:50.731012 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:50.730977 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60409783-fdb7-4b2e-9ce3-05fd3be2ab8c-webhook-cert\") pod \"maas-controller-868857dbd9-6b89z\" (UID: \"60409783-fdb7-4b2e-9ce3-05fd3be2ab8c\") " pod="opendatahub/maas-controller-868857dbd9-6b89z" Nov 04 03:23:50.731184 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:50.731044 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6m7z\" (UniqueName: \"kubernetes.io/projected/60409783-fdb7-4b2e-9ce3-05fd3be2ab8c-kube-api-access-l6m7z\") pod \"maas-controller-868857dbd9-6b89z\" (UID: \"60409783-fdb7-4b2e-9ce3-05fd3be2ab8c\") " pod="opendatahub/maas-controller-868857dbd9-6b89z" Nov 04 03:23:50.731184 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:50.731144 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60409783-fdb7-4b2e-9ce3-05fd3be2ab8c-metrics-certs\") pod \"maas-controller-868857dbd9-6b89z\" (UID: \"60409783-fdb7-4b2e-9ce3-05fd3be2ab8c\") " pod="opendatahub/maas-controller-868857dbd9-6b89z" Nov 04 03:23:50.832601 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:50.832564 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6m7z\" (UniqueName: \"kubernetes.io/projected/60409783-fdb7-4b2e-9ce3-05fd3be2ab8c-kube-api-access-l6m7z\") pod \"maas-controller-868857dbd9-6b89z\" (UID: \"60409783-fdb7-4b2e-9ce3-05fd3be2ab8c\") " pod="opendatahub/maas-controller-868857dbd9-6b89z" Nov 04 03:23:50.832793 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:50.832614 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60409783-fdb7-4b2e-9ce3-05fd3be2ab8c-metrics-certs\") pod \"maas-controller-868857dbd9-6b89z\" (UID: \"60409783-fdb7-4b2e-9ce3-05fd3be2ab8c\") " pod="opendatahub/maas-controller-868857dbd9-6b89z" Nov 04 03:23:50.832793 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:50.832689 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60409783-fdb7-4b2e-9ce3-05fd3be2ab8c-webhook-cert\") pod \"maas-controller-868857dbd9-6b89z\" (UID: \"60409783-fdb7-4b2e-9ce3-05fd3be2ab8c\") " pod="opendatahub/maas-controller-868857dbd9-6b89z" Nov 04 03:23:50.835111 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:50.835079 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60409783-fdb7-4b2e-9ce3-05fd3be2ab8c-webhook-cert\") pod \"maas-controller-868857dbd9-6b89z\" (UID: \"60409783-fdb7-4b2e-9ce3-05fd3be2ab8c\") " pod="opendatahub/maas-controller-868857dbd9-6b89z" Nov 04 03:23:50.835219 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:50.835146 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60409783-fdb7-4b2e-9ce3-05fd3be2ab8c-metrics-certs\") pod \"maas-controller-868857dbd9-6b89z\" (UID: \"60409783-fdb7-4b2e-9ce3-05fd3be2ab8c\") " pod="opendatahub/maas-controller-868857dbd9-6b89z" Nov 04 03:23:50.895222 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:50.895147 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6m7z\" (UniqueName: \"kubernetes.io/projected/60409783-fdb7-4b2e-9ce3-05fd3be2ab8c-kube-api-access-l6m7z\") pod \"maas-controller-868857dbd9-6b89z\" (UID: \"60409783-fdb7-4b2e-9ce3-05fd3be2ab8c\") " pod="opendatahub/maas-controller-868857dbd9-6b89z" Nov 04 03:23:51.193999 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:51.193889 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-868857dbd9-6b89z" Nov 04 03:23:51.564936 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:51.564858 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-868857dbd9-6b89z"] Nov 04 03:23:51.599511 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:23:51.599479 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60409783_fdb7_4b2e_9ce3_05fd3be2ab8c.slice/crio-9f5906d8054aeabcc23952869ef6873ae3127accdd4bf68384a5d27d377bd922 WatchSource:0}: Error finding container 9f5906d8054aeabcc23952869ef6873ae3127accdd4bf68384a5d27d377bd922: Status 404 returned error can't find the container with id 9f5906d8054aeabcc23952869ef6873ae3127accdd4bf68384a5d27d377bd922 Nov 04 03:23:52.433198 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:52.433160 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-868857dbd9-6b89z" event={"ID":"60409783-fdb7-4b2e-9ce3-05fd3be2ab8c","Type":"ContainerStarted","Data":"9f5906d8054aeabcc23952869ef6873ae3127accdd4bf68384a5d27d377bd922"} Nov 04 03:23:59.080790 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:59.080762 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 04 03:23:59.470487 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:59.470392 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-868857dbd9-6b89z" event={"ID":"60409783-fdb7-4b2e-9ce3-05fd3be2ab8c","Type":"ContainerStarted","Data":"399520f694b7d35ab448988627326b56ef1f7ebd2754e7869eca9b1d53517a27"} Nov 04 03:23:59.470678 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:59.470486 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-868857dbd9-6b89z" Nov 04 03:23:59.588983 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:23:59.588926 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-868857dbd9-6b89z" podStartSLOduration=2.161569773 podStartE2EDuration="9.588911183s" totalImagesPullingTime="7.42734141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:23:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:23:59.587891493 +0000 UTC m=+364.862522255" watchObservedRunningTime="2026-11-04 03:23:59.588911183 +0000 UTC m=+364.863541943" Nov 04 03:24:02.797100 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:02.797067 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tt4m7"] Nov 04 03:24:02.800660 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:02.800639 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tt4m7" Nov 04 03:24:02.853414 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:02.853388 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 04 03:24:02.853414 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:02.853396 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 04 03:24:02.903862 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:02.903831 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 04 03:24:02.904225 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:02.904206 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-r4dkc\"" Nov 04 03:24:02.905280 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:02.905257 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tt4m7"] Nov 04 03:24:02.950599 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:02.950562 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e53b6123-3a57-4a27-87e4-d430f8d8e4cc-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tt4m7\" (UID: \"e53b6123-3a57-4a27-87e4-d430f8d8e4cc\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tt4m7" Nov 04 03:24:02.950833 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:02.950658 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/e53b6123-3a57-4a27-87e4-d430f8d8e4cc-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tt4m7\" (UID: \"e53b6123-3a57-4a27-87e4-d430f8d8e4cc\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tt4m7" Nov 04 03:24:02.950833 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:02.950697 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e53b6123-3a57-4a27-87e4-d430f8d8e4cc-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tt4m7\" (UID: \"e53b6123-3a57-4a27-87e4-d430f8d8e4cc\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tt4m7" Nov 04 03:24:02.950833 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:02.950794 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9xjq\" (UniqueName: \"kubernetes.io/projected/e53b6123-3a57-4a27-87e4-d430f8d8e4cc-kube-api-access-c9xjq\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tt4m7\" (UID: \"e53b6123-3a57-4a27-87e4-d430f8d8e4cc\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tt4m7" Nov 04 03:24:03.052002 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:03.051906 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e53b6123-3a57-4a27-87e4-d430f8d8e4cc-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tt4m7\" (UID: \"e53b6123-3a57-4a27-87e4-d430f8d8e4cc\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tt4m7" Nov 04 03:24:03.052002 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:03.051978 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/e53b6123-3a57-4a27-87e4-d430f8d8e4cc-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tt4m7\" (UID: \"e53b6123-3a57-4a27-87e4-d430f8d8e4cc\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tt4m7" Nov 04 03:24:03.052243 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:03.052023 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e53b6123-3a57-4a27-87e4-d430f8d8e4cc-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tt4m7\" (UID: \"e53b6123-3a57-4a27-87e4-d430f8d8e4cc\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tt4m7" Nov 04 03:24:03.052243 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:03.052048 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9xjq\" (UniqueName: \"kubernetes.io/projected/e53b6123-3a57-4a27-87e4-d430f8d8e4cc-kube-api-access-c9xjq\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tt4m7\" (UID: \"e53b6123-3a57-4a27-87e4-d430f8d8e4cc\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tt4m7" Nov 04 03:24:03.052243 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:24:03.052167 5953 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Nov 04 03:24:03.052243 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:24:03.052245 5953 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e53b6123-3a57-4a27-87e4-d430f8d8e4cc-proxy-source podName:e53b6123-3a57-4a27-87e4-d430f8d8e4cc nodeName:}" failed. No retries permitted until 2026-11-04 03:24:03.55222261 +0000 UTC m=+368.826853368 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/e53b6123-3a57-4a27-87e4-d430f8d8e4cc-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-tt4m7" (UID: "e53b6123-3a57-4a27-87e4-d430f8d8e4cc") : configmap "usage-tenancy-proxy-source" not found Nov 04 03:24:03.052609 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:03.052589 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e53b6123-3a57-4a27-87e4-d430f8d8e4cc-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tt4m7\" (UID: \"e53b6123-3a57-4a27-87e4-d430f8d8e4cc\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tt4m7" Nov 04 03:24:03.054489 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:03.054468 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e53b6123-3a57-4a27-87e4-d430f8d8e4cc-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tt4m7\" (UID: \"e53b6123-3a57-4a27-87e4-d430f8d8e4cc\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tt4m7" Nov 04 03:24:03.114493 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:03.114461 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9xjq\" (UniqueName: \"kubernetes.io/projected/e53b6123-3a57-4a27-87e4-d430f8d8e4cc-kube-api-access-c9xjq\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tt4m7\" (UID: \"e53b6123-3a57-4a27-87e4-d430f8d8e4cc\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tt4m7" Nov 04 03:24:03.556771 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:03.556732 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/e53b6123-3a57-4a27-87e4-d430f8d8e4cc-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tt4m7\" (UID: \"e53b6123-3a57-4a27-87e4-d430f8d8e4cc\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tt4m7" Nov 04 03:24:03.557534 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:03.557514 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/e53b6123-3a57-4a27-87e4-d430f8d8e4cc-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tt4m7\" (UID: \"e53b6123-3a57-4a27-87e4-d430f8d8e4cc\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tt4m7" Nov 04 03:24:03.713541 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:03.713463 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tt4m7" Nov 04 03:24:04.087532 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:04.087498 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tt4m7"] Nov 04 03:24:04.090892 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:24:04.090865 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode53b6123_3a57_4a27_87e4_d430f8d8e4cc.slice/crio-9b141e8596dfa14d811f0741b631b4cae909422d167a1d9caa651b3c59c003a7 WatchSource:0}: Error finding container 9b141e8596dfa14d811f0741b631b4cae909422d167a1d9caa651b3c59c003a7: Status 404 returned error can't find the container with id 9b141e8596dfa14d811f0741b631b4cae909422d167a1d9caa651b3c59c003a7 Nov 04 03:24:04.496782 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:04.496704 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tt4m7" event={"ID":"e53b6123-3a57-4a27-87e4-d430f8d8e4cc","Type":"ContainerStarted","Data":"9b141e8596dfa14d811f0741b631b4cae909422d167a1d9caa651b3c59c003a7"} Nov 04 03:24:10.478421 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:10.477543 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-868857dbd9-6b89z" Nov 04 03:24:16.390496 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:24:16.390455 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:24:17.575267 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:17.575231 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tt4m7" event={"ID":"e53b6123-3a57-4a27-87e4-d430f8d8e4cc","Type":"ContainerStarted","Data":"19e5a2fac24f4767e4a5041829381461963314eec07f25811c58fa853f5848e7"} Nov 04 03:24:17.575906 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:17.575372 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tt4m7" Nov 04 03:24:17.693756 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:17.693706 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tt4m7" podStartSLOduration=3.109618021 podStartE2EDuration="15.693691953s" totalImagesPullingTime="12.584073932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:24:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:24:17.692516499 +0000 UTC m=+382.967147260" watchObservedRunningTime="2026-11-04 03:24:17.693691953 +0000 UTC m=+382.968322708" Nov 04 03:24:28.585913 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:28.585882 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tt4m7" Nov 04 03:24:36.302995 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:36.302955 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6b496f7748-s757v"] Nov 04 03:24:36.307135 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:36.307114 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b496f7748-s757v" Nov 04 03:24:36.362067 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:36.362037 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:24:36.412219 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:36.412181 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-m6mdd\"" Nov 04 03:24:36.412219 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:36.412210 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:24:36.412431 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:36.412212 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 04 03:24:36.412431 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:36.412288 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 04 03:24:36.413611 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:36.413591 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b496f7748-s757v"] Nov 04 03:24:36.474940 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:36.474903 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e819bcdb-e976-4458-bc4e-40b8d2f69d6d-maas-api-tls\") pod \"maas-api-6b496f7748-s757v\" (UID: \"e819bcdb-e976-4458-bc4e-40b8d2f69d6d\") " pod="odh-ai-gateway-infra/maas-api-6b496f7748-s757v" Nov 04 03:24:36.475114 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:36.474945 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e819bcdb-e976-4458-bc4e-40b8d2f69d6d-openshift-service-ca\") pod \"maas-api-6b496f7748-s757v\" (UID: \"e819bcdb-e976-4458-bc4e-40b8d2f69d6d\") " pod="odh-ai-gateway-infra/maas-api-6b496f7748-s757v" Nov 04 03:24:36.475114 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:36.475039 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-887qd\" (UniqueName: \"kubernetes.io/projected/e819bcdb-e976-4458-bc4e-40b8d2f69d6d-kube-api-access-887qd\") pod \"maas-api-6b496f7748-s757v\" (UID: \"e819bcdb-e976-4458-bc4e-40b8d2f69d6d\") " pod="odh-ai-gateway-infra/maas-api-6b496f7748-s757v" Nov 04 03:24:36.475206 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:36.475136 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e819bcdb-e976-4458-bc4e-40b8d2f69d6d-metrics-certs\") pod \"maas-api-6b496f7748-s757v\" (UID: \"e819bcdb-e976-4458-bc4e-40b8d2f69d6d\") " pod="odh-ai-gateway-infra/maas-api-6b496f7748-s757v" Nov 04 03:24:36.575939 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:36.575855 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e819bcdb-e976-4458-bc4e-40b8d2f69d6d-maas-api-tls\") pod \"maas-api-6b496f7748-s757v\" (UID: \"e819bcdb-e976-4458-bc4e-40b8d2f69d6d\") " pod="odh-ai-gateway-infra/maas-api-6b496f7748-s757v" Nov 04 03:24:36.575939 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:36.575902 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e819bcdb-e976-4458-bc4e-40b8d2f69d6d-openshift-service-ca\") pod \"maas-api-6b496f7748-s757v\" (UID: \"e819bcdb-e976-4458-bc4e-40b8d2f69d6d\") " pod="odh-ai-gateway-infra/maas-api-6b496f7748-s757v" Nov 04 03:24:36.576122 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:36.576085 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-887qd\" (UniqueName: \"kubernetes.io/projected/e819bcdb-e976-4458-bc4e-40b8d2f69d6d-kube-api-access-887qd\") pod \"maas-api-6b496f7748-s757v\" (UID: \"e819bcdb-e976-4458-bc4e-40b8d2f69d6d\") " pod="odh-ai-gateway-infra/maas-api-6b496f7748-s757v" Nov 04 03:24:36.576159 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:36.576139 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e819bcdb-e976-4458-bc4e-40b8d2f69d6d-metrics-certs\") pod \"maas-api-6b496f7748-s757v\" (UID: \"e819bcdb-e976-4458-bc4e-40b8d2f69d6d\") " pod="odh-ai-gateway-infra/maas-api-6b496f7748-s757v" Nov 04 03:24:36.576481 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:36.576448 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e819bcdb-e976-4458-bc4e-40b8d2f69d6d-openshift-service-ca\") pod \"maas-api-6b496f7748-s757v\" (UID: \"e819bcdb-e976-4458-bc4e-40b8d2f69d6d\") " pod="odh-ai-gateway-infra/maas-api-6b496f7748-s757v" Nov 04 03:24:36.578559 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:36.578538 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e819bcdb-e976-4458-bc4e-40b8d2f69d6d-metrics-certs\") pod \"maas-api-6b496f7748-s757v\" (UID: \"e819bcdb-e976-4458-bc4e-40b8d2f69d6d\") " pod="odh-ai-gateway-infra/maas-api-6b496f7748-s757v" Nov 04 03:24:36.578662 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:36.578645 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e819bcdb-e976-4458-bc4e-40b8d2f69d6d-maas-api-tls\") pod \"maas-api-6b496f7748-s757v\" (UID: \"e819bcdb-e976-4458-bc4e-40b8d2f69d6d\") " pod="odh-ai-gateway-infra/maas-api-6b496f7748-s757v" Nov 04 03:24:36.637576 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:36.637543 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-887qd\" (UniqueName: \"kubernetes.io/projected/e819bcdb-e976-4458-bc4e-40b8d2f69d6d-kube-api-access-887qd\") pod \"maas-api-6b496f7748-s757v\" (UID: \"e819bcdb-e976-4458-bc4e-40b8d2f69d6d\") " pod="odh-ai-gateway-infra/maas-api-6b496f7748-s757v" Nov 04 03:24:36.921590 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:36.921510 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b496f7748-s757v" Nov 04 03:24:37.293997 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:37.293961 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b496f7748-s757v"] Nov 04 03:24:37.297209 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:24:37.297185 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode819bcdb_e976_4458_bc4e_40b8d2f69d6d.slice/crio-caf7ff9314e20f93a8d969201515806d0152ea58693a5960b2a2897710314408 WatchSource:0}: Error finding container caf7ff9314e20f93a8d969201515806d0152ea58693a5960b2a2897710314408: Status 404 returned error can't find the container with id caf7ff9314e20f93a8d969201515806d0152ea58693a5960b2a2897710314408 Nov 04 03:24:37.666778 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:37.666697 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b496f7748-s757v" event={"ID":"e819bcdb-e976-4458-bc4e-40b8d2f69d6d","Type":"ContainerStarted","Data":"caf7ff9314e20f93a8d969201515806d0152ea58693a5960b2a2897710314408"} Nov 04 03:24:44.705735 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:44.705694 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b496f7748-s757v" event={"ID":"e819bcdb-e976-4458-bc4e-40b8d2f69d6d","Type":"ContainerStarted","Data":"6f597e44fd72c899e4a78f52876b486bba1d808dbba3e5b9e858f24f17b5f2e5"} Nov 04 03:24:44.706158 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:44.705778 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6b496f7748-s757v" Nov 04 03:24:44.824888 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:44.824831 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6b496f7748-s757v" podStartSLOduration=2.462275602 podStartE2EDuration="8.824815288s" totalImagesPullingTime="6.362539686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:24:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:24:44.823889622 +0000 UTC m=+410.098520386" watchObservedRunningTime="2026-11-04 03:24:44.824815288 +0000 UTC m=+410.099446050" Nov 04 03:24:46.384598 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:24:46.384556 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:24:50.715650 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:50.715589 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6b496f7748-s757v" Nov 04 03:24:52.046752 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:52.046721 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-648d4dbc8d-djsnr"] Nov 04 03:24:52.078251 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:52.078220 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-648d4dbc8d-djsnr" Nov 04 03:24:52.112240 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:52.112208 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-648d4dbc8d-djsnr"] Nov 04 03:24:52.121980 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:52.121957 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klz5j\" (UniqueName: \"kubernetes.io/projected/4a691860-e8f3-44e7-b7b7-547534680415-kube-api-access-klz5j\") pod \"authorino-648d4dbc8d-djsnr\" (UID: \"4a691860-e8f3-44e7-b7b7-547534680415\") " pod="kuadrant-system/authorino-648d4dbc8d-djsnr" Nov 04 03:24:52.222443 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:52.222415 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klz5j\" (UniqueName: \"kubernetes.io/projected/4a691860-e8f3-44e7-b7b7-547534680415-kube-api-access-klz5j\") pod \"authorino-648d4dbc8d-djsnr\" (UID: \"4a691860-e8f3-44e7-b7b7-547534680415\") " pod="kuadrant-system/authorino-648d4dbc8d-djsnr" Nov 04 03:24:52.284048 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:52.284020 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klz5j\" (UniqueName: \"kubernetes.io/projected/4a691860-e8f3-44e7-b7b7-547534680415-kube-api-access-klz5j\") pod \"authorino-648d4dbc8d-djsnr\" (UID: \"4a691860-e8f3-44e7-b7b7-547534680415\") " pod="kuadrant-system/authorino-648d4dbc8d-djsnr" Nov 04 03:24:52.388447 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:52.388375 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-648d4dbc8d-djsnr" Nov 04 03:24:52.453527 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:52.453181 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-648d4dbc8d-djsnr"] Nov 04 03:24:52.754534 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:52.754505 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-648d4dbc8d-djsnr"] Nov 04 03:24:52.757794 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:24:52.757765 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a691860_e8f3_44e7_b7b7_547534680415.slice/crio-f02eaf6d0f21df904b63144dab85f173664a2118d29b451f1c3c3b833b52d4a4 WatchSource:0}: Error finding container f02eaf6d0f21df904b63144dab85f173664a2118d29b451f1c3c3b833b52d4a4: Status 404 returned error can't find the container with id f02eaf6d0f21df904b63144dab85f173664a2118d29b451f1c3c3b833b52d4a4 Nov 04 03:24:53.746685 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:53.746640 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-648d4dbc8d-djsnr" event={"ID":"4a691860-e8f3-44e7-b7b7-547534680415","Type":"ContainerStarted","Data":"f02eaf6d0f21df904b63144dab85f173664a2118d29b451f1c3c3b833b52d4a4"} Nov 04 03:24:54.754770 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:54.754731 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-648d4dbc8d-djsnr" event={"ID":"4a691860-e8f3-44e7-b7b7-547534680415","Type":"ContainerStarted","Data":"77f61e3ea514d497f50070ec2273ab9174edf4aa057830a40a8deeb608401278"} Nov 04 03:24:54.755311 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:54.754808 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-648d4dbc8d-djsnr" podUID="4a691860-e8f3-44e7-b7b7-547534680415" containerName="authorino" containerID="cri-o://77f61e3ea514d497f50070ec2273ab9174edf4aa057830a40a8deeb608401278" gracePeriod=30 Nov 04 03:24:54.880703 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:54.878960 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-648d4dbc8d-djsnr" podStartSLOduration=1.561109875 podStartE2EDuration="2.878942981s" totalImagesPullingTime="1.317833106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:24:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:24:54.877945842 +0000 UTC m=+420.152576605" watchObservedRunningTime="2026-11-04 03:24:54.878942981 +0000 UTC m=+420.153573743" Nov 04 03:24:55.006973 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:55.006925 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-648d4dbc8d-djsnr" Nov 04 03:24:55.050654 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:55.050611 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a691860-e8f3-44e7-b7b7-547534680415-kube-api-access-klz5j\" (UniqueName: \"kubernetes.io/projected/4a691860-e8f3-44e7-b7b7-547534680415-kube-api-access-klz5j\") pod \"4a691860-e8f3-44e7-b7b7-547534680415\" (UID: \"4a691860-e8f3-44e7-b7b7-547534680415\") " Nov 04 03:24:55.052779 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:55.052756 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a691860-e8f3-44e7-b7b7-547534680415-kube-api-access-klz5j" pod "4a691860-e8f3-44e7-b7b7-547534680415" (UID: "4a691860-e8f3-44e7-b7b7-547534680415"). InnerVolumeSpecName "kube-api-access-klz5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:24:55.152211 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:55.152167 5953 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klz5j\" (UniqueName: \"kubernetes.io/projected/4a691860-e8f3-44e7-b7b7-547534680415-kube-api-access-klz5j\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:24:55.493087 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:55.493013 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-4ldrz"] Nov 04 03:24:55.493210 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:55.493186 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-4ldrz" podUID="bd5cdc80-d609-4312-89c7-226da9334e2f" containerName="authorino" containerID="cri-o://f5b2b82891422224c6d9d7183b2345c4094d2e7d03e138955f7b99bbfee3b7b6" gracePeriod=30 Nov 04 03:24:55.744164 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:55.744111 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-4ldrz" Nov 04 03:24:55.760001 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:55.759973 5953 generic.go:356] "Generic (PLEG): container finished" podID="4a691860-e8f3-44e7-b7b7-547534680415" containerID="77f61e3ea514d497f50070ec2273ab9174edf4aa057830a40a8deeb608401278" exitCode=0 Nov 04 03:24:55.760353 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:55.760019 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-648d4dbc8d-djsnr" Nov 04 03:24:55.760353 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:55.760048 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-648d4dbc8d-djsnr" event={"ID":"4a691860-e8f3-44e7-b7b7-547534680415","Type":"ContainerDied","Data":"77f61e3ea514d497f50070ec2273ab9174edf4aa057830a40a8deeb608401278"} Nov 04 03:24:55.760353 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:55.760094 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-648d4dbc8d-djsnr" event={"ID":"4a691860-e8f3-44e7-b7b7-547534680415","Type":"ContainerDied","Data":"f02eaf6d0f21df904b63144dab85f173664a2118d29b451f1c3c3b833b52d4a4"} Nov 04 03:24:55.760353 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:55.760112 5953 scope.go:122] "RemoveContainer" containerID="77f61e3ea514d497f50070ec2273ab9174edf4aa057830a40a8deeb608401278" Nov 04 03:24:55.761695 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:55.761677 5953 generic.go:356] "Generic (PLEG): container finished" podID="bd5cdc80-d609-4312-89c7-226da9334e2f" containerID="f5b2b82891422224c6d9d7183b2345c4094d2e7d03e138955f7b99bbfee3b7b6" exitCode=0 Nov 04 03:24:55.761795 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:55.761713 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-4ldrz" event={"ID":"bd5cdc80-d609-4312-89c7-226da9334e2f","Type":"ContainerDied","Data":"f5b2b82891422224c6d9d7183b2345c4094d2e7d03e138955f7b99bbfee3b7b6"} Nov 04 03:24:55.761795 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:55.761736 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-4ldrz" event={"ID":"bd5cdc80-d609-4312-89c7-226da9334e2f","Type":"ContainerDied","Data":"26852e2ea59663c357e1212605dd75a4c4ce4004639ea58850475a1f931be486"} Nov 04 03:24:55.761795 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:55.761748 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-4ldrz" Nov 04 03:24:55.768690 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:55.768676 5953 scope.go:122] "RemoveContainer" containerID="77f61e3ea514d497f50070ec2273ab9174edf4aa057830a40a8deeb608401278" Nov 04 03:24:55.768967 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:24:55.768949 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77f61e3ea514d497f50070ec2273ab9174edf4aa057830a40a8deeb608401278\": container with ID starting with 77f61e3ea514d497f50070ec2273ab9174edf4aa057830a40a8deeb608401278 not found: ID does not exist" containerID="77f61e3ea514d497f50070ec2273ab9174edf4aa057830a40a8deeb608401278" Nov 04 03:24:55.769030 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:55.768977 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77f61e3ea514d497f50070ec2273ab9174edf4aa057830a40a8deeb608401278"} err="failed to get container status \"77f61e3ea514d497f50070ec2273ab9174edf4aa057830a40a8deeb608401278\": rpc error: code = NotFound desc = could not find container \"77f61e3ea514d497f50070ec2273ab9174edf4aa057830a40a8deeb608401278\": container with ID starting with 77f61e3ea514d497f50070ec2273ab9174edf4aa057830a40a8deeb608401278 not found: ID does not exist" Nov 04 03:24:55.769030 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:55.768994 5953 scope.go:122] "RemoveContainer" containerID="f5b2b82891422224c6d9d7183b2345c4094d2e7d03e138955f7b99bbfee3b7b6" Nov 04 03:24:55.776340 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:55.776324 5953 scope.go:122] "RemoveContainer" containerID="f5b2b82891422224c6d9d7183b2345c4094d2e7d03e138955f7b99bbfee3b7b6" Nov 04 03:24:55.776593 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:24:55.776575 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5b2b82891422224c6d9d7183b2345c4094d2e7d03e138955f7b99bbfee3b7b6\": container with ID starting with f5b2b82891422224c6d9d7183b2345c4094d2e7d03e138955f7b99bbfee3b7b6 not found: ID does not exist" containerID="f5b2b82891422224c6d9d7183b2345c4094d2e7d03e138955f7b99bbfee3b7b6" Nov 04 03:24:55.776667 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:55.776599 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5b2b82891422224c6d9d7183b2345c4094d2e7d03e138955f7b99bbfee3b7b6"} err="failed to get container status \"f5b2b82891422224c6d9d7183b2345c4094d2e7d03e138955f7b99bbfee3b7b6\": rpc error: code = NotFound desc = could not find container \"f5b2b82891422224c6d9d7183b2345c4094d2e7d03e138955f7b99bbfee3b7b6\": container with ID starting with f5b2b82891422224c6d9d7183b2345c4094d2e7d03e138955f7b99bbfee3b7b6 not found: ID does not exist" Nov 04 03:24:55.856870 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:55.856847 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd5cdc80-d609-4312-89c7-226da9334e2f-kube-api-access-rzdrs\" (UniqueName: \"kubernetes.io/projected/bd5cdc80-d609-4312-89c7-226da9334e2f-kube-api-access-rzdrs\") pod \"bd5cdc80-d609-4312-89c7-226da9334e2f\" (UID: \"bd5cdc80-d609-4312-89c7-226da9334e2f\") " Nov 04 03:24:55.858864 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:55.858835 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd5cdc80-d609-4312-89c7-226da9334e2f-kube-api-access-rzdrs" pod "bd5cdc80-d609-4312-89c7-226da9334e2f" (UID: "bd5cdc80-d609-4312-89c7-226da9334e2f"). InnerVolumeSpecName "kube-api-access-rzdrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:24:55.957772 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:55.957747 5953 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzdrs\" (UniqueName: \"kubernetes.io/projected/bd5cdc80-d609-4312-89c7-226da9334e2f-kube-api-access-rzdrs\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:24:56.024228 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:56.024203 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-648d4dbc8d-djsnr"] Nov 04 03:24:56.028723 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:56.028702 5953 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-648d4dbc8d-djsnr"] Nov 04 03:24:56.183768 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:56.183740 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-4ldrz"] Nov 04 03:24:56.187982 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:56.187953 5953 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-4ldrz"] Nov 04 03:24:56.854322 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:56.854290 5953 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a691860-e8f3-44e7-b7b7-547534680415" path="/var/lib/kubelet/pods/4a691860-e8f3-44e7-b7b7-547534680415/volumes" Nov 04 03:24:56.854700 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:24:56.854585 5953 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd5cdc80-d609-4312-89c7-226da9334e2f" path="/var/lib/kubelet/pods/bd5cdc80-d609-4312-89c7-226da9334e2f/volumes" Nov 04 03:25:03.513422 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:03.513388 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b496f7748-s757v"] Nov 04 03:25:03.513839 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:03.513607 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6b496f7748-s757v" podUID="e819bcdb-e976-4458-bc4e-40b8d2f69d6d" containerName="maas-api" containerID="cri-o://6f597e44fd72c899e4a78f52876b486bba1d808dbba3e5b9e858f24f17b5f2e5" gracePeriod=30 Nov 04 03:25:03.751197 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:03.751174 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b496f7748-s757v" Nov 04 03:25:03.800659 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:03.800584 5953 generic.go:356] "Generic (PLEG): container finished" podID="e819bcdb-e976-4458-bc4e-40b8d2f69d6d" containerID="6f597e44fd72c899e4a78f52876b486bba1d808dbba3e5b9e858f24f17b5f2e5" exitCode=0 Nov 04 03:25:03.800774 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:03.800656 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b496f7748-s757v" Nov 04 03:25:03.800774 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:03.800682 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b496f7748-s757v" event={"ID":"e819bcdb-e976-4458-bc4e-40b8d2f69d6d","Type":"ContainerDied","Data":"6f597e44fd72c899e4a78f52876b486bba1d808dbba3e5b9e858f24f17b5f2e5"} Nov 04 03:25:03.800774 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:03.800740 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b496f7748-s757v" event={"ID":"e819bcdb-e976-4458-bc4e-40b8d2f69d6d","Type":"ContainerDied","Data":"caf7ff9314e20f93a8d969201515806d0152ea58693a5960b2a2897710314408"} Nov 04 03:25:03.800774 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:03.800761 5953 scope.go:122] "RemoveContainer" containerID="6f597e44fd72c899e4a78f52876b486bba1d808dbba3e5b9e858f24f17b5f2e5" Nov 04 03:25:03.809058 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:03.809035 5953 scope.go:122] "RemoveContainer" containerID="6f597e44fd72c899e4a78f52876b486bba1d808dbba3e5b9e858f24f17b5f2e5" Nov 04 03:25:03.809334 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:25:03.809317 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f597e44fd72c899e4a78f52876b486bba1d808dbba3e5b9e858f24f17b5f2e5\": container with ID starting with 6f597e44fd72c899e4a78f52876b486bba1d808dbba3e5b9e858f24f17b5f2e5 not found: ID does not exist" containerID="6f597e44fd72c899e4a78f52876b486bba1d808dbba3e5b9e858f24f17b5f2e5" Nov 04 03:25:03.809387 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:03.809343 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f597e44fd72c899e4a78f52876b486bba1d808dbba3e5b9e858f24f17b5f2e5"} err="failed to get container status \"6f597e44fd72c899e4a78f52876b486bba1d808dbba3e5b9e858f24f17b5f2e5\": rpc error: code = NotFound desc = could not find container \"6f597e44fd72c899e4a78f52876b486bba1d808dbba3e5b9e858f24f17b5f2e5\": container with ID starting with 6f597e44fd72c899e4a78f52876b486bba1d808dbba3e5b9e858f24f17b5f2e5 not found: ID does not exist" Nov 04 03:25:03.824655 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:03.824612 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e819bcdb-e976-4458-bc4e-40b8d2f69d6d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e819bcdb-e976-4458-bc4e-40b8d2f69d6d-maas-api-tls\") pod \"e819bcdb-e976-4458-bc4e-40b8d2f69d6d\" (UID: \"e819bcdb-e976-4458-bc4e-40b8d2f69d6d\") " Nov 04 03:25:03.824730 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:03.824667 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e819bcdb-e976-4458-bc4e-40b8d2f69d6d-kube-api-access-887qd\" (UniqueName: \"kubernetes.io/projected/e819bcdb-e976-4458-bc4e-40b8d2f69d6d-kube-api-access-887qd\") pod \"e819bcdb-e976-4458-bc4e-40b8d2f69d6d\" (UID: \"e819bcdb-e976-4458-bc4e-40b8d2f69d6d\") " Nov 04 03:25:03.824805 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:03.824793 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e819bcdb-e976-4458-bc4e-40b8d2f69d6d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e819bcdb-e976-4458-bc4e-40b8d2f69d6d-openshift-service-ca\") pod \"e819bcdb-e976-4458-bc4e-40b8d2f69d6d\" (UID: \"e819bcdb-e976-4458-bc4e-40b8d2f69d6d\") " Nov 04 03:25:03.824849 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:03.824813 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e819bcdb-e976-4458-bc4e-40b8d2f69d6d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e819bcdb-e976-4458-bc4e-40b8d2f69d6d-metrics-certs\") pod \"e819bcdb-e976-4458-bc4e-40b8d2f69d6d\" (UID: \"e819bcdb-e976-4458-bc4e-40b8d2f69d6d\") " Nov 04 03:25:03.825137 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:03.825110 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e819bcdb-e976-4458-bc4e-40b8d2f69d6d-openshift-service-ca" pod "e819bcdb-e976-4458-bc4e-40b8d2f69d6d" (UID: "e819bcdb-e976-4458-bc4e-40b8d2f69d6d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:25:03.826785 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:03.826754 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e819bcdb-e976-4458-bc4e-40b8d2f69d6d-maas-api-tls" pod "e819bcdb-e976-4458-bc4e-40b8d2f69d6d" (UID: "e819bcdb-e976-4458-bc4e-40b8d2f69d6d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:25:03.826785 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:03.826774 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e819bcdb-e976-4458-bc4e-40b8d2f69d6d-kube-api-access-887qd" pod "e819bcdb-e976-4458-bc4e-40b8d2f69d6d" (UID: "e819bcdb-e976-4458-bc4e-40b8d2f69d6d"). InnerVolumeSpecName "kube-api-access-887qd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:25:03.826922 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:03.826824 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e819bcdb-e976-4458-bc4e-40b8d2f69d6d-metrics-certs" pod "e819bcdb-e976-4458-bc4e-40b8d2f69d6d" (UID: "e819bcdb-e976-4458-bc4e-40b8d2f69d6d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:25:03.925841 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:03.925813 5953 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e819bcdb-e976-4458-bc4e-40b8d2f69d6d-openshift-service-ca\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:25:03.925841 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:03.925837 5953 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e819bcdb-e976-4458-bc4e-40b8d2f69d6d-metrics-certs\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:25:03.925841 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:03.925846 5953 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e819bcdb-e976-4458-bc4e-40b8d2f69d6d-maas-api-tls\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:25:03.926046 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:03.925854 5953 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-887qd\" (UniqueName: \"kubernetes.io/projected/e819bcdb-e976-4458-bc4e-40b8d2f69d6d-kube-api-access-887qd\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:25:04.220973 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:04.220946 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b496f7748-s757v"] Nov 04 03:25:04.226364 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:04.226342 5953 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b496f7748-s757v"] Nov 04 03:25:04.854073 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:04.854039 5953 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e819bcdb-e976-4458-bc4e-40b8d2f69d6d" path="/var/lib/kubelet/pods/e819bcdb-e976-4458-bc4e-40b8d2f69d6d/volumes" Nov 04 03:25:16.384389 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:25:16.384354 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:25:31.806826 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:31.806785 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-rw8nw"] Nov 04 03:25:31.807373 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:31.807343 5953 cpu_manager.go:414] "containerMap: removing container" podUID="4a691860-e8f3-44e7-b7b7-547534680415" containerName="authorino" Nov 04 03:25:31.807480 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:31.807375 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a691860-e8f3-44e7-b7b7-547534680415" containerName="authorino" Nov 04 03:25:31.807480 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:31.807414 5953 cpu_manager.go:414] "containerMap: removing container" podUID="bd5cdc80-d609-4312-89c7-226da9334e2f" containerName="authorino" Nov 04 03:25:31.807480 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:31.807421 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd5cdc80-d609-4312-89c7-226da9334e2f" containerName="authorino" Nov 04 03:25:31.807480 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:31.807455 5953 cpu_manager.go:414] "containerMap: removing container" podUID="e819bcdb-e976-4458-bc4e-40b8d2f69d6d" containerName="maas-api" Nov 04 03:25:31.807480 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:31.807463 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e819bcdb-e976-4458-bc4e-40b8d2f69d6d" containerName="maas-api" Nov 04 03:25:31.807760 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:31.807581 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="e819bcdb-e976-4458-bc4e-40b8d2f69d6d" containerName="maas-api" Nov 04 03:25:31.807760 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:31.807600 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a691860-e8f3-44e7-b7b7-547534680415" containerName="authorino" Nov 04 03:25:31.807760 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:31.807611 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd5cdc80-d609-4312-89c7-226da9334e2f" containerName="authorino" Nov 04 03:25:31.811535 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:31.811512 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-rw8nw" Nov 04 03:25:31.868680 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:31.868637 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bcba100-9bee-4c9b-97c5-826a40add62e-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-rw8nw\" (UID: \"9bcba100-9bee-4c9b-97c5-826a40add62e\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-rw8nw" Nov 04 03:25:31.868866 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:31.868689 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2qms\" (UniqueName: \"kubernetes.io/projected/9bcba100-9bee-4c9b-97c5-826a40add62e-kube-api-access-n2qms\") pod \"ai-gateway-controller-5d8dddb676-rw8nw\" (UID: \"9bcba100-9bee-4c9b-97c5-826a40add62e\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-rw8nw" Nov 04 03:25:31.917552 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:31.917501 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 04 03:25:31.917552 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:31.917539 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 04 03:25:31.917840 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:31.917594 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-sh8zp\"" Nov 04 03:25:31.918751 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:31.918726 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-rw8nw"] Nov 04 03:25:31.969170 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:31.969130 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bcba100-9bee-4c9b-97c5-826a40add62e-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-rw8nw\" (UID: \"9bcba100-9bee-4c9b-97c5-826a40add62e\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-rw8nw" Nov 04 03:25:31.969170 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:31.969172 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2qms\" (UniqueName: \"kubernetes.io/projected/9bcba100-9bee-4c9b-97c5-826a40add62e-kube-api-access-n2qms\") pod \"ai-gateway-controller-5d8dddb676-rw8nw\" (UID: \"9bcba100-9bee-4c9b-97c5-826a40add62e\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-rw8nw" Nov 04 03:25:31.971763 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:31.971738 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bcba100-9bee-4c9b-97c5-826a40add62e-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-rw8nw\" (UID: \"9bcba100-9bee-4c9b-97c5-826a40add62e\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-rw8nw" Nov 04 03:25:32.031984 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:32.031942 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2qms\" (UniqueName: \"kubernetes.io/projected/9bcba100-9bee-4c9b-97c5-826a40add62e-kube-api-access-n2qms\") pod \"ai-gateway-controller-5d8dddb676-rw8nw\" (UID: \"9bcba100-9bee-4c9b-97c5-826a40add62e\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-rw8nw" Nov 04 03:25:32.124343 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:32.124247 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-rw8nw" Nov 04 03:25:32.499532 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:32.499439 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-rw8nw"] Nov 04 03:25:32.502417 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:25:32.502391 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bcba100_9bee_4c9b_97c5_826a40add62e.slice/crio-9103aede00f8b4e9da5884f0223495107f61168eea35737a8a1b05b8e79f1f71 WatchSource:0}: Error finding container 9103aede00f8b4e9da5884f0223495107f61168eea35737a8a1b05b8e79f1f71: Status 404 returned error can't find the container with id 9103aede00f8b4e9da5884f0223495107f61168eea35737a8a1b05b8e79f1f71 Nov 04 03:25:32.927992 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:32.927958 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-rw8nw" event={"ID":"9bcba100-9bee-4c9b-97c5-826a40add62e","Type":"ContainerStarted","Data":"9103aede00f8b4e9da5884f0223495107f61168eea35737a8a1b05b8e79f1f71"} Nov 04 03:25:34.939492 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:34.939378 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-rw8nw" event={"ID":"9bcba100-9bee-4c9b-97c5-826a40add62e","Type":"ContainerStarted","Data":"6495fba3c48e5c0e87c7f6a2b953cfdc274759e0c09cca43cd55786afd3a9502"} Nov 04 03:25:34.939976 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:34.939506 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-rw8nw" Nov 04 03:25:35.057165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:35.057112 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d8dddb676-rw8nw" podStartSLOduration=1.94601405 podStartE2EDuration="4.057097962s" totalImagesPullingTime="2.111083912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:25:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:25:35.056342327 +0000 UTC m=+460.330973088" watchObservedRunningTime="2026-11-04 03:25:35.057097962 +0000 UTC m=+460.331728774" Nov 04 03:25:37.731309 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:37.731275 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-86fmg"] Nov 04 03:25:37.734753 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:37.734735 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-86fmg" Nov 04 03:25:37.787769 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:37.787736 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 04 03:25:37.825252 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:37.825211 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jklsm\" (UniqueName: \"kubernetes.io/projected/bab1308d-3935-4601-acb9-48ac9ba1cdc9-kube-api-access-jklsm\") pod \"payload-pre-processing-7b98bb678c-86fmg\" (UID: \"bab1308d-3935-4601-acb9-48ac9ba1cdc9\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-86fmg" Nov 04 03:25:37.825252 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:37.825251 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bab1308d-3935-4601-acb9-48ac9ba1cdc9-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-86fmg\" (UID: \"bab1308d-3935-4601-acb9-48ac9ba1cdc9\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-86fmg" Nov 04 03:25:37.838216 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:37.838186 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-gdlcb\"" Nov 04 03:25:37.839276 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:37.839255 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-86fmg"] Nov 04 03:25:37.875539 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:37.875505 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-vzvct"] Nov 04 03:25:37.878916 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:37.878899 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-vzvct" Nov 04 03:25:37.926772 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:37.926740 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/050f4e90-30e5-4f04-9293-63424357799f-plugins-config-volume\") pod \"payload-processing-7bb9489676-vzvct\" (UID: \"050f4e90-30e5-4f04-9293-63424357799f\") " pod="openshift-ingress/payload-processing-7bb9489676-vzvct" Nov 04 03:25:37.926772 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:37.926775 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jklsm\" (UniqueName: \"kubernetes.io/projected/bab1308d-3935-4601-acb9-48ac9ba1cdc9-kube-api-access-jklsm\") pod \"payload-pre-processing-7b98bb678c-86fmg\" (UID: \"bab1308d-3935-4601-acb9-48ac9ba1cdc9\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-86fmg" Nov 04 03:25:37.926987 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:37.926795 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bab1308d-3935-4601-acb9-48ac9ba1cdc9-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-86fmg\" (UID: \"bab1308d-3935-4601-acb9-48ac9ba1cdc9\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-86fmg" Nov 04 03:25:37.926987 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:37.926893 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gklt\" (UniqueName: \"kubernetes.io/projected/050f4e90-30e5-4f04-9293-63424357799f-kube-api-access-7gklt\") pod \"payload-processing-7bb9489676-vzvct\" (UID: \"050f4e90-30e5-4f04-9293-63424357799f\") " pod="openshift-ingress/payload-processing-7bb9489676-vzvct" Nov 04 03:25:37.927435 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:37.927415 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bab1308d-3935-4601-acb9-48ac9ba1cdc9-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-86fmg\" (UID: \"bab1308d-3935-4601-acb9-48ac9ba1cdc9\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-86fmg" Nov 04 03:25:37.936523 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:37.936482 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-vzvct"] Nov 04 03:25:37.988571 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:37.988504 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jklsm\" (UniqueName: \"kubernetes.io/projected/bab1308d-3935-4601-acb9-48ac9ba1cdc9-kube-api-access-jklsm\") pod \"payload-pre-processing-7b98bb678c-86fmg\" (UID: \"bab1308d-3935-4601-acb9-48ac9ba1cdc9\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-86fmg" Nov 04 03:25:38.027667 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:38.027609 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/050f4e90-30e5-4f04-9293-63424357799f-plugins-config-volume\") pod \"payload-processing-7bb9489676-vzvct\" (UID: \"050f4e90-30e5-4f04-9293-63424357799f\") " pod="openshift-ingress/payload-processing-7bb9489676-vzvct" Nov 04 03:25:38.027838 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:38.027679 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gklt\" (UniqueName: \"kubernetes.io/projected/050f4e90-30e5-4f04-9293-63424357799f-kube-api-access-7gklt\") pod \"payload-processing-7bb9489676-vzvct\" (UID: \"050f4e90-30e5-4f04-9293-63424357799f\") " pod="openshift-ingress/payload-processing-7bb9489676-vzvct" Nov 04 03:25:38.028384 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:38.028362 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/050f4e90-30e5-4f04-9293-63424357799f-plugins-config-volume\") pod \"payload-processing-7bb9489676-vzvct\" (UID: \"050f4e90-30e5-4f04-9293-63424357799f\") " pod="openshift-ingress/payload-processing-7bb9489676-vzvct" Nov 04 03:25:38.046461 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:38.046431 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-86fmg" Nov 04 03:25:38.090478 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:38.090445 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gklt\" (UniqueName: \"kubernetes.io/projected/050f4e90-30e5-4f04-9293-63424357799f-kube-api-access-7gklt\") pod \"payload-processing-7bb9489676-vzvct\" (UID: \"050f4e90-30e5-4f04-9293-63424357799f\") " pod="openshift-ingress/payload-processing-7bb9489676-vzvct" Nov 04 03:25:38.192203 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:38.192170 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-vzvct" Nov 04 03:25:38.421281 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:38.421247 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-86fmg"] Nov 04 03:25:38.424450 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:25:38.424423 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbab1308d_3935_4601_acb9_48ac9ba1cdc9.slice/crio-1d93c0ca2b9cf8f94830feb36d76bb6d4621f0fd23b4f8c3bb2157083bae1a5e WatchSource:0}: Error finding container 1d93c0ca2b9cf8f94830feb36d76bb6d4621f0fd23b4f8c3bb2157083bae1a5e: Status 404 returned error can't find the container with id 1d93c0ca2b9cf8f94830feb36d76bb6d4621f0fd23b4f8c3bb2157083bae1a5e Nov 04 03:25:38.589939 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:38.589904 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-vzvct"] Nov 04 03:25:38.592985 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:25:38.592963 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod050f4e90_30e5_4f04_9293_63424357799f.slice/crio-d1344eb30174819314619f25409d6c7346d16428fc810c98c5299e1e24812a24 WatchSource:0}: Error finding container d1344eb30174819314619f25409d6c7346d16428fc810c98c5299e1e24812a24: Status 404 returned error can't find the container with id d1344eb30174819314619f25409d6c7346d16428fc810c98c5299e1e24812a24 Nov 04 03:25:38.960060 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:38.959968 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-vzvct" event={"ID":"050f4e90-30e5-4f04-9293-63424357799f","Type":"ContainerStarted","Data":"d1344eb30174819314619f25409d6c7346d16428fc810c98c5299e1e24812a24"} Nov 04 03:25:38.961082 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:38.961062 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-86fmg" event={"ID":"bab1308d-3935-4601-acb9-48ac9ba1cdc9","Type":"ContainerStarted","Data":"1d93c0ca2b9cf8f94830feb36d76bb6d4621f0fd23b4f8c3bb2157083bae1a5e"} Nov 04 03:25:41.982151 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:41.982110 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-vzvct" event={"ID":"050f4e90-30e5-4f04-9293-63424357799f","Type":"ContainerStarted","Data":"e67d4c50e67f84c47128fce66873666e015d40a2a7c0a9b28418c5f41d54c8c2"} Nov 04 03:25:41.982657 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:41.982272 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-7bb9489676-vzvct" Nov 04 03:25:41.983787 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:41.983763 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-86fmg" event={"ID":"bab1308d-3935-4601-acb9-48ac9ba1cdc9","Type":"ContainerStarted","Data":"0945b89ecfbab29e54364f76d541353be7fd8fac19b71538888d3d81e006cb60"} Nov 04 03:25:41.983889 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:41.983813 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-86fmg" Nov 04 03:25:42.099672 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:42.099597 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-7bb9489676-vzvct" podStartSLOduration=2.293938025 podStartE2EDuration="5.099582115s" totalImagesPullingTime="2.80564409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:25:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:25:42.098081762 +0000 UTC m=+467.372712524" watchObservedRunningTime="2026-11-04 03:25:42.099582115 +0000 UTC m=+467.374212921" Nov 04 03:25:45.947041 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:45.947010 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-rw8nw" Nov 04 03:25:46.064817 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:46.064760 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-86fmg" podStartSLOduration=6.056868303 podStartE2EDuration="9.064743715s" totalImagesPullingTime="3.007875412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:25:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:25:42.215517396 +0000 UTC m=+467.490148159" watchObservedRunningTime="2026-11-04 03:25:46.064743715 +0000 UTC m=+471.339374475" Nov 04 03:25:46.304303 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:46.304269 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-d69f556c7-qx7mw"] Nov 04 03:25:46.308021 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:46.308001 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-d69f556c7-qx7mw" Nov 04 03:25:46.366454 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:46.366420 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-d69f556c7-qx7mw"] Nov 04 03:25:46.386518 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:25:46.386477 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:25:46.410973 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:46.410940 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60f0eec5-a469-42ad-8f4f-d11a9b840081-metrics-certs\") pod \"ai-gateway-controller-d69f556c7-qx7mw\" (UID: \"60f0eec5-a469-42ad-8f4f-d11a9b840081\") " pod="opendatahub/ai-gateway-controller-d69f556c7-qx7mw" Nov 04 03:25:46.411118 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:46.411093 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc7rx\" (UniqueName: \"kubernetes.io/projected/60f0eec5-a469-42ad-8f4f-d11a9b840081-kube-api-access-jc7rx\") pod \"ai-gateway-controller-d69f556c7-qx7mw\" (UID: \"60f0eec5-a469-42ad-8f4f-d11a9b840081\") " pod="opendatahub/ai-gateway-controller-d69f556c7-qx7mw" Nov 04 03:25:46.511682 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:46.511643 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60f0eec5-a469-42ad-8f4f-d11a9b840081-metrics-certs\") pod \"ai-gateway-controller-d69f556c7-qx7mw\" (UID: \"60f0eec5-a469-42ad-8f4f-d11a9b840081\") " pod="opendatahub/ai-gateway-controller-d69f556c7-qx7mw" Nov 04 03:25:46.511877 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:46.511728 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc7rx\" (UniqueName: \"kubernetes.io/projected/60f0eec5-a469-42ad-8f4f-d11a9b840081-kube-api-access-jc7rx\") pod \"ai-gateway-controller-d69f556c7-qx7mw\" (UID: \"60f0eec5-a469-42ad-8f4f-d11a9b840081\") " pod="opendatahub/ai-gateway-controller-d69f556c7-qx7mw" Nov 04 03:25:46.514172 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:46.514149 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60f0eec5-a469-42ad-8f4f-d11a9b840081-metrics-certs\") pod \"ai-gateway-controller-d69f556c7-qx7mw\" (UID: \"60f0eec5-a469-42ad-8f4f-d11a9b840081\") " pod="opendatahub/ai-gateway-controller-d69f556c7-qx7mw" Nov 04 03:25:46.573687 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:46.573593 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc7rx\" (UniqueName: \"kubernetes.io/projected/60f0eec5-a469-42ad-8f4f-d11a9b840081-kube-api-access-jc7rx\") pod \"ai-gateway-controller-d69f556c7-qx7mw\" (UID: \"60f0eec5-a469-42ad-8f4f-d11a9b840081\") " pod="opendatahub/ai-gateway-controller-d69f556c7-qx7mw" Nov 04 03:25:46.620682 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:46.620644 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-d69f556c7-qx7mw" Nov 04 03:25:46.993341 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:46.993265 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-d69f556c7-qx7mw"] Nov 04 03:25:46.996347 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:25:46.996314 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f0eec5_a469_42ad_8f4f_d11a9b840081.slice/crio-d900c2ce35c50de9a04e5a4e3c7d9b8035dfcb40e2c6cb19cb7edc94798f9ffb WatchSource:0}: Error finding container d900c2ce35c50de9a04e5a4e3c7d9b8035dfcb40e2c6cb19cb7edc94798f9ffb: Status 404 returned error can't find the container with id d900c2ce35c50de9a04e5a4e3c7d9b8035dfcb40e2c6cb19cb7edc94798f9ffb Nov 04 03:25:47.008383 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:47.008358 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-d69f556c7-qx7mw" event={"ID":"60f0eec5-a469-42ad-8f4f-d11a9b840081","Type":"ContainerStarted","Data":"d900c2ce35c50de9a04e5a4e3c7d9b8035dfcb40e2c6cb19cb7edc94798f9ffb"} Nov 04 03:25:48.014295 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:48.014259 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-d69f556c7-qx7mw" event={"ID":"60f0eec5-a469-42ad-8f4f-d11a9b840081","Type":"ContainerStarted","Data":"efcaab632360d747a5dace81a9f6e0991b38d4c2639a068d72135bf190552aa6"} Nov 04 03:25:48.014838 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:48.014381 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-d69f556c7-qx7mw" Nov 04 03:25:48.133117 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:48.133014 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-d69f556c7-qx7mw" podStartSLOduration=1.318768604 podStartE2EDuration="2.1329989s" totalImagesPullingTime="814.230296ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:25:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:25:48.131484639 +0000 UTC m=+473.406115402" watchObservedRunningTime="2026-11-04 03:25:48.1329989 +0000 UTC m=+473.407629716" Nov 04 03:25:52.991988 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:52.991958 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-7bb9489676-vzvct" Nov 04 03:25:52.992400 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:52.992029 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-86fmg" Nov 04 03:25:59.021433 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:59.021402 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-d69f556c7-qx7mw" Nov 04 03:25:59.188525 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:59.188479 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-rw8nw"] Nov 04 03:25:59.188772 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:59.188747 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d8dddb676-rw8nw" podUID="9bcba100-9bee-4c9b-97c5-826a40add62e" containerName="manager" containerID="cri-o://6495fba3c48e5c0e87c7f6a2b953cfdc274759e0c09cca43cd55786afd3a9502" gracePeriod=30 Nov 04 03:25:59.436045 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:59.436016 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-rw8nw" Nov 04 03:25:59.531561 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:59.531526 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bcba100-9bee-4c9b-97c5-826a40add62e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bcba100-9bee-4c9b-97c5-826a40add62e-metrics-certs\") pod \"9bcba100-9bee-4c9b-97c5-826a40add62e\" (UID: \"9bcba100-9bee-4c9b-97c5-826a40add62e\") " Nov 04 03:25:59.531766 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:59.531581 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bcba100-9bee-4c9b-97c5-826a40add62e-kube-api-access-n2qms\" (UniqueName: \"kubernetes.io/projected/9bcba100-9bee-4c9b-97c5-826a40add62e-kube-api-access-n2qms\") pod \"9bcba100-9bee-4c9b-97c5-826a40add62e\" (UID: \"9bcba100-9bee-4c9b-97c5-826a40add62e\") " Nov 04 03:25:59.533993 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:59.533966 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bcba100-9bee-4c9b-97c5-826a40add62e-kube-api-access-n2qms" pod "9bcba100-9bee-4c9b-97c5-826a40add62e" (UID: "9bcba100-9bee-4c9b-97c5-826a40add62e"). InnerVolumeSpecName "kube-api-access-n2qms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:25:59.534053 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:59.534000 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bcba100-9bee-4c9b-97c5-826a40add62e-metrics-certs" pod "9bcba100-9bee-4c9b-97c5-826a40add62e" (UID: "9bcba100-9bee-4c9b-97c5-826a40add62e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:25:59.633244 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:59.633147 5953 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bcba100-9bee-4c9b-97c5-826a40add62e-metrics-certs\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:25:59.633244 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:25:59.633187 5953 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2qms\" (UniqueName: \"kubernetes.io/projected/9bcba100-9bee-4c9b-97c5-826a40add62e-kube-api-access-n2qms\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:26:00.077024 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:00.076992 5953 generic.go:356] "Generic (PLEG): container finished" podID="9bcba100-9bee-4c9b-97c5-826a40add62e" containerID="6495fba3c48e5c0e87c7f6a2b953cfdc274759e0c09cca43cd55786afd3a9502" exitCode=0 Nov 04 03:26:00.077496 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:00.077051 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-rw8nw" Nov 04 03:26:00.077496 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:00.077073 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-rw8nw" event={"ID":"9bcba100-9bee-4c9b-97c5-826a40add62e","Type":"ContainerDied","Data":"6495fba3c48e5c0e87c7f6a2b953cfdc274759e0c09cca43cd55786afd3a9502"} Nov 04 03:26:00.077496 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:00.077170 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-rw8nw" event={"ID":"9bcba100-9bee-4c9b-97c5-826a40add62e","Type":"ContainerDied","Data":"9103aede00f8b4e9da5884f0223495107f61168eea35737a8a1b05b8e79f1f71"} Nov 04 03:26:00.077496 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:00.077202 5953 scope.go:122] "RemoveContainer" containerID="6495fba3c48e5c0e87c7f6a2b953cfdc274759e0c09cca43cd55786afd3a9502" Nov 04 03:26:00.086301 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:00.086284 5953 scope.go:122] "RemoveContainer" containerID="6495fba3c48e5c0e87c7f6a2b953cfdc274759e0c09cca43cd55786afd3a9502" Nov 04 03:26:00.086581 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:26:00.086564 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6495fba3c48e5c0e87c7f6a2b953cfdc274759e0c09cca43cd55786afd3a9502\": container with ID starting with 6495fba3c48e5c0e87c7f6a2b953cfdc274759e0c09cca43cd55786afd3a9502 not found: ID does not exist" containerID="6495fba3c48e5c0e87c7f6a2b953cfdc274759e0c09cca43cd55786afd3a9502" Nov 04 03:26:00.086653 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:00.086590 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6495fba3c48e5c0e87c7f6a2b953cfdc274759e0c09cca43cd55786afd3a9502"} err="failed to get container status \"6495fba3c48e5c0e87c7f6a2b953cfdc274759e0c09cca43cd55786afd3a9502\": rpc error: code = NotFound desc = could not find container \"6495fba3c48e5c0e87c7f6a2b953cfdc274759e0c09cca43cd55786afd3a9502\": container with ID starting with 6495fba3c48e5c0e87c7f6a2b953cfdc274759e0c09cca43cd55786afd3a9502 not found: ID does not exist" Nov 04 03:26:00.244074 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:00.244036 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-rw8nw"] Nov 04 03:26:00.248056 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:00.248028 5953 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-rw8nw"] Nov 04 03:26:00.854510 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:00.854484 5953 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bcba100-9bee-4c9b-97c5-826a40add62e" path="/var/lib/kubelet/pods/9bcba100-9bee-4c9b-97c5-826a40add62e/volumes" Nov 04 03:26:09.158209 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:09.158172 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg"] Nov 04 03:26:09.158596 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:09.158580 5953 cpu_manager.go:414] "containerMap: removing container" podUID="9bcba100-9bee-4c9b-97c5-826a40add62e" containerName="manager" Nov 04 03:26:09.158596 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:09.158590 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bcba100-9bee-4c9b-97c5-826a40add62e" containerName="manager" Nov 04 03:26:09.158743 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:09.158731 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bcba100-9bee-4c9b-97c5-826a40add62e" containerName="manager" Nov 04 03:26:09.161989 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:09.161969 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg" Nov 04 03:26:09.236878 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:09.236850 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 04 03:26:09.236878 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:09.236879 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8fl6l\"" Nov 04 03:26:09.265076 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:09.265042 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 04 03:26:09.265076 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:09.265069 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 04 03:26:09.266455 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:09.266434 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg"] Nov 04 03:26:09.322773 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:09.322739 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83e293dd-236c-4b48-a841-27dd9190db27-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg\" (UID: \"83e293dd-236c-4b48-a841-27dd9190db27\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg" Nov 04 03:26:09.322945 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:09.322806 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83e293dd-236c-4b48-a841-27dd9190db27-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg\" (UID: \"83e293dd-236c-4b48-a841-27dd9190db27\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg" Nov 04 03:26:09.322945 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:09.322880 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83e293dd-236c-4b48-a841-27dd9190db27-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg\" (UID: \"83e293dd-236c-4b48-a841-27dd9190db27\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg" Nov 04 03:26:09.322945 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:09.322910 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83e293dd-236c-4b48-a841-27dd9190db27-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg\" (UID: \"83e293dd-236c-4b48-a841-27dd9190db27\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg" Nov 04 03:26:09.323065 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:09.323009 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83e293dd-236c-4b48-a841-27dd9190db27-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg\" (UID: \"83e293dd-236c-4b48-a841-27dd9190db27\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg" Nov 04 03:26:09.323065 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:09.323039 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttbzc\" (UniqueName: \"kubernetes.io/projected/83e293dd-236c-4b48-a841-27dd9190db27-kube-api-access-ttbzc\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg\" (UID: \"83e293dd-236c-4b48-a841-27dd9190db27\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg" Nov 04 03:26:09.423760 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:09.423685 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83e293dd-236c-4b48-a841-27dd9190db27-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg\" (UID: \"83e293dd-236c-4b48-a841-27dd9190db27\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg" Nov 04 03:26:09.423760 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:09.423725 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttbzc\" (UniqueName: \"kubernetes.io/projected/83e293dd-236c-4b48-a841-27dd9190db27-kube-api-access-ttbzc\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg\" (UID: \"83e293dd-236c-4b48-a841-27dd9190db27\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg" Nov 04 03:26:09.423989 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:09.423769 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83e293dd-236c-4b48-a841-27dd9190db27-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg\" (UID: \"83e293dd-236c-4b48-a841-27dd9190db27\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg" Nov 04 03:26:09.423989 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:09.423824 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83e293dd-236c-4b48-a841-27dd9190db27-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg\" (UID: \"83e293dd-236c-4b48-a841-27dd9190db27\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg" Nov 04 03:26:09.423989 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:09.423842 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83e293dd-236c-4b48-a841-27dd9190db27-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg\" (UID: \"83e293dd-236c-4b48-a841-27dd9190db27\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg" Nov 04 03:26:09.423989 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:09.423862 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83e293dd-236c-4b48-a841-27dd9190db27-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg\" (UID: \"83e293dd-236c-4b48-a841-27dd9190db27\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg" Nov 04 03:26:09.424200 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:09.424174 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83e293dd-236c-4b48-a841-27dd9190db27-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg\" (UID: \"83e293dd-236c-4b48-a841-27dd9190db27\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg" Nov 04 03:26:09.424264 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:09.424223 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83e293dd-236c-4b48-a841-27dd9190db27-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg\" (UID: \"83e293dd-236c-4b48-a841-27dd9190db27\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg" Nov 04 03:26:09.424314 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:09.424261 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83e293dd-236c-4b48-a841-27dd9190db27-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg\" (UID: \"83e293dd-236c-4b48-a841-27dd9190db27\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg" Nov 04 03:26:09.426183 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:09.426163 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83e293dd-236c-4b48-a841-27dd9190db27-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg\" (UID: \"83e293dd-236c-4b48-a841-27dd9190db27\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg" Nov 04 03:26:09.426411 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:09.426395 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83e293dd-236c-4b48-a841-27dd9190db27-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg\" (UID: \"83e293dd-236c-4b48-a841-27dd9190db27\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg" Nov 04 03:26:09.487755 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:09.487721 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttbzc\" (UniqueName: \"kubernetes.io/projected/83e293dd-236c-4b48-a841-27dd9190db27-kube-api-access-ttbzc\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg\" (UID: \"83e293dd-236c-4b48-a841-27dd9190db27\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg" Nov 04 03:26:09.774457 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:09.774417 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg" Nov 04 03:26:10.149153 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:10.149085 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg"] Nov 04 03:26:10.152687 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:26:10.152656 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83e293dd_236c_4b48_a841_27dd9190db27.slice/crio-b7beef10a944f234b1078f335de81952adada2167bbfdd8e35d5083ce910bc76 WatchSource:0}: Error finding container b7beef10a944f234b1078f335de81952adada2167bbfdd8e35d5083ce910bc76: Status 404 returned error can't find the container with id b7beef10a944f234b1078f335de81952adada2167bbfdd8e35d5083ce910bc76 Nov 04 03:26:11.133589 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:11.133539 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg" event={"ID":"83e293dd-236c-4b48-a841-27dd9190db27","Type":"ContainerStarted","Data":"b7beef10a944f234b1078f335de81952adada2167bbfdd8e35d5083ce910bc76"} Nov 04 03:26:13.144823 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:13.144781 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg" event={"ID":"83e293dd-236c-4b48-a841-27dd9190db27","Type":"ContainerStarted","Data":"dbf09adb3a71dac25b0339fe3ea7bf14f34f7f4ad115491075fd068e36fac28c"} Nov 04 03:26:13.264327 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:13.264268 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg" podStartSLOduration=2.061296209 podStartE2EDuration="4.264253124s" totalImagesPullingTime="2.202956915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:26:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:26:13.263500594 +0000 UTC m=+498.538131356" watchObservedRunningTime="2026-11-04 03:26:13.264253124 +0000 UTC m=+498.538883885" Nov 04 03:26:13.774848 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:13.774810 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg" Nov 04 03:26:16.386856 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:26:16.386809 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:26:17.766816 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:17.766777 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-cmqzr"] Nov 04 03:26:17.772245 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:17.772219 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmqzr" Nov 04 03:26:17.790656 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:17.790629 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51318cd3-19e0-40ca-8731-78f8b0d18a26-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cmqzr\" (UID: \"51318cd3-19e0-40ca-8731-78f8b0d18a26\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmqzr" Nov 04 03:26:17.790901 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:17.790735 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51318cd3-19e0-40ca-8731-78f8b0d18a26-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cmqzr\" (UID: \"51318cd3-19e0-40ca-8731-78f8b0d18a26\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmqzr" Nov 04 03:26:17.790901 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:17.790780 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgxsn\" (UniqueName: \"kubernetes.io/projected/51318cd3-19e0-40ca-8731-78f8b0d18a26-kube-api-access-hgxsn\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cmqzr\" (UID: \"51318cd3-19e0-40ca-8731-78f8b0d18a26\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmqzr" Nov 04 03:26:17.790901 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:17.790806 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51318cd3-19e0-40ca-8731-78f8b0d18a26-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cmqzr\" (UID: \"51318cd3-19e0-40ca-8731-78f8b0d18a26\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmqzr" Nov 04 03:26:17.790901 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:17.790822 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51318cd3-19e0-40ca-8731-78f8b0d18a26-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cmqzr\" (UID: \"51318cd3-19e0-40ca-8731-78f8b0d18a26\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmqzr" Nov 04 03:26:17.791048 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:17.790920 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51318cd3-19e0-40ca-8731-78f8b0d18a26-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cmqzr\" (UID: \"51318cd3-19e0-40ca-8731-78f8b0d18a26\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmqzr" Nov 04 03:26:17.878291 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:17.878257 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 04 03:26:17.879579 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:17.879551 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-cmqzr"] Nov 04 03:26:17.892611 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:17.892577 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51318cd3-19e0-40ca-8731-78f8b0d18a26-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cmqzr\" (UID: \"51318cd3-19e0-40ca-8731-78f8b0d18a26\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmqzr" Nov 04 03:26:17.892881 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:17.892859 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgxsn\" (UniqueName: \"kubernetes.io/projected/51318cd3-19e0-40ca-8731-78f8b0d18a26-kube-api-access-hgxsn\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cmqzr\" (UID: \"51318cd3-19e0-40ca-8731-78f8b0d18a26\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmqzr" Nov 04 03:26:17.892972 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:17.892925 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51318cd3-19e0-40ca-8731-78f8b0d18a26-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cmqzr\" (UID: \"51318cd3-19e0-40ca-8731-78f8b0d18a26\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmqzr" Nov 04 03:26:17.893029 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:17.892970 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51318cd3-19e0-40ca-8731-78f8b0d18a26-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cmqzr\" (UID: \"51318cd3-19e0-40ca-8731-78f8b0d18a26\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmqzr" Nov 04 03:26:17.893081 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:17.893053 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51318cd3-19e0-40ca-8731-78f8b0d18a26-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cmqzr\" (UID: \"51318cd3-19e0-40ca-8731-78f8b0d18a26\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmqzr" Nov 04 03:26:17.893132 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:17.893112 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51318cd3-19e0-40ca-8731-78f8b0d18a26-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cmqzr\" (UID: \"51318cd3-19e0-40ca-8731-78f8b0d18a26\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmqzr" Nov 04 03:26:17.893611 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:17.893442 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51318cd3-19e0-40ca-8731-78f8b0d18a26-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cmqzr\" (UID: \"51318cd3-19e0-40ca-8731-78f8b0d18a26\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmqzr" Nov 04 03:26:17.893611 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:17.893552 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51318cd3-19e0-40ca-8731-78f8b0d18a26-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cmqzr\" (UID: \"51318cd3-19e0-40ca-8731-78f8b0d18a26\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmqzr" Nov 04 03:26:17.893783 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:17.893701 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51318cd3-19e0-40ca-8731-78f8b0d18a26-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cmqzr\" (UID: \"51318cd3-19e0-40ca-8731-78f8b0d18a26\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmqzr" Nov 04 03:26:17.895314 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:17.895292 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51318cd3-19e0-40ca-8731-78f8b0d18a26-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cmqzr\" (UID: \"51318cd3-19e0-40ca-8731-78f8b0d18a26\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmqzr" Nov 04 03:26:17.895512 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:17.895492 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51318cd3-19e0-40ca-8731-78f8b0d18a26-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cmqzr\" (UID: \"51318cd3-19e0-40ca-8731-78f8b0d18a26\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmqzr" Nov 04 03:26:17.954663 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:17.954614 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgxsn\" (UniqueName: \"kubernetes.io/projected/51318cd3-19e0-40ca-8731-78f8b0d18a26-kube-api-access-hgxsn\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cmqzr\" (UID: \"51318cd3-19e0-40ca-8731-78f8b0d18a26\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmqzr" Nov 04 03:26:18.084209 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:18.084122 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmqzr" Nov 04 03:26:18.458262 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:18.458178 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-cmqzr"] Nov 04 03:26:18.461502 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:26:18.461478 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51318cd3_19e0_40ca_8731_78f8b0d18a26.slice/crio-ecadd254683a25014c1aa38c2f633929816f261eb1fb456084539608c7130c95 WatchSource:0}: Error finding container ecadd254683a25014c1aa38c2f633929816f261eb1fb456084539608c7130c95: Status 404 returned error can't find the container with id ecadd254683a25014c1aa38c2f633929816f261eb1fb456084539608c7130c95 Nov 04 03:26:19.175075 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:19.175038 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmqzr" event={"ID":"51318cd3-19e0-40ca-8731-78f8b0d18a26","Type":"ContainerStarted","Data":"1e592e64cf2bd8db53ac3117003a8aceb8b280dc98deda85ead19dec294ba0bc"} Nov 04 03:26:19.175533 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:19.175114 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmqzr" event={"ID":"51318cd3-19e0-40ca-8731-78f8b0d18a26","Type":"ContainerStarted","Data":"ecadd254683a25014c1aa38c2f633929816f261eb1fb456084539608c7130c95"} Nov 04 03:26:19.297598 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:19.297544 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmqzr" podStartSLOduration=1.7031908329999998 podStartE2EDuration="2.297528378s" totalImagesPullingTime="594.337545ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:26:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:26:19.296469212 +0000 UTC m=+504.571099974" watchObservedRunningTime="2026-11-04 03:26:19.297528378 +0000 UTC m=+504.572159140" Nov 04 03:26:19.774959 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:19.774925 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg" Nov 04 03:26:19.787756 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:19.787730 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg" Nov 04 03:26:20.084777 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:20.084692 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmqzr" Nov 04 03:26:20.191892 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:20.191865 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg" Nov 04 03:26:21.698854 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:21.698818 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-ftccw"] Nov 04 03:26:21.702494 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:21.702476 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ftccw" Nov 04 03:26:21.730297 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:21.730270 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a08f9b6-140f-4f6b-974a-dcce183d79c8-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ftccw\" (UID: \"5a08f9b6-140f-4f6b-974a-dcce183d79c8\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ftccw" Nov 04 03:26:21.730463 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:21.730306 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a08f9b6-140f-4f6b-974a-dcce183d79c8-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ftccw\" (UID: \"5a08f9b6-140f-4f6b-974a-dcce183d79c8\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ftccw" Nov 04 03:26:21.730463 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:21.730350 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltdpj\" (UniqueName: \"kubernetes.io/projected/5a08f9b6-140f-4f6b-974a-dcce183d79c8-kube-api-access-ltdpj\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ftccw\" (UID: \"5a08f9b6-140f-4f6b-974a-dcce183d79c8\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ftccw" Nov 04 03:26:21.730463 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:21.730449 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a08f9b6-140f-4f6b-974a-dcce183d79c8-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ftccw\" (UID: \"5a08f9b6-140f-4f6b-974a-dcce183d79c8\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ftccw" Nov 04 03:26:21.730563 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:21.730471 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a08f9b6-140f-4f6b-974a-dcce183d79c8-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ftccw\" (UID: \"5a08f9b6-140f-4f6b-974a-dcce183d79c8\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ftccw" Nov 04 03:26:21.730563 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:21.730491 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a08f9b6-140f-4f6b-974a-dcce183d79c8-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ftccw\" (UID: \"5a08f9b6-140f-4f6b-974a-dcce183d79c8\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ftccw" Nov 04 03:26:21.755396 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:21.755361 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 04 03:26:21.807040 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:21.807007 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-ftccw"] Nov 04 03:26:21.832041 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:21.832011 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a08f9b6-140f-4f6b-974a-dcce183d79c8-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ftccw\" (UID: \"5a08f9b6-140f-4f6b-974a-dcce183d79c8\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ftccw" Nov 04 03:26:21.832235 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:21.832070 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltdpj\" (UniqueName: \"kubernetes.io/projected/5a08f9b6-140f-4f6b-974a-dcce183d79c8-kube-api-access-ltdpj\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ftccw\" (UID: \"5a08f9b6-140f-4f6b-974a-dcce183d79c8\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ftccw" Nov 04 03:26:21.832235 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:21.832097 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a08f9b6-140f-4f6b-974a-dcce183d79c8-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ftccw\" (UID: \"5a08f9b6-140f-4f6b-974a-dcce183d79c8\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ftccw" Nov 04 03:26:21.832235 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:21.832115 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a08f9b6-140f-4f6b-974a-dcce183d79c8-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ftccw\" (UID: \"5a08f9b6-140f-4f6b-974a-dcce183d79c8\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ftccw" Nov 04 03:26:21.832235 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:21.832131 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a08f9b6-140f-4f6b-974a-dcce183d79c8-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ftccw\" (UID: \"5a08f9b6-140f-4f6b-974a-dcce183d79c8\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ftccw" Nov 04 03:26:21.832235 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:21.832172 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a08f9b6-140f-4f6b-974a-dcce183d79c8-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ftccw\" (UID: \"5a08f9b6-140f-4f6b-974a-dcce183d79c8\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ftccw" Nov 04 03:26:21.832509 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:21.832481 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a08f9b6-140f-4f6b-974a-dcce183d79c8-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ftccw\" (UID: \"5a08f9b6-140f-4f6b-974a-dcce183d79c8\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ftccw" Nov 04 03:26:21.832564 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:21.832491 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a08f9b6-140f-4f6b-974a-dcce183d79c8-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ftccw\" (UID: \"5a08f9b6-140f-4f6b-974a-dcce183d79c8\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ftccw" Nov 04 03:26:21.832611 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:21.832559 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a08f9b6-140f-4f6b-974a-dcce183d79c8-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ftccw\" (UID: \"5a08f9b6-140f-4f6b-974a-dcce183d79c8\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ftccw" Nov 04 03:26:21.834563 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:21.834537 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a08f9b6-140f-4f6b-974a-dcce183d79c8-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ftccw\" (UID: \"5a08f9b6-140f-4f6b-974a-dcce183d79c8\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ftccw" Nov 04 03:26:21.834760 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:21.834744 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a08f9b6-140f-4f6b-974a-dcce183d79c8-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ftccw\" (UID: \"5a08f9b6-140f-4f6b-974a-dcce183d79c8\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ftccw" Nov 04 03:26:21.894258 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:21.894220 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltdpj\" (UniqueName: \"kubernetes.io/projected/5a08f9b6-140f-4f6b-974a-dcce183d79c8-kube-api-access-ltdpj\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-ftccw\" (UID: \"5a08f9b6-140f-4f6b-974a-dcce183d79c8\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ftccw" Nov 04 03:26:22.015381 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:22.015323 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ftccw" Nov 04 03:26:22.388520 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:22.388425 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-ftccw"] Nov 04 03:26:22.391532 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:26:22.391507 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a08f9b6_140f_4f6b_974a_dcce183d79c8.slice/crio-657009e88c98148895c754316e6eb4823d5cf033a0ee11d26e0cf67eeef80d3e WatchSource:0}: Error finding container 657009e88c98148895c754316e6eb4823d5cf033a0ee11d26e0cf67eeef80d3e: Status 404 returned error can't find the container with id 657009e88c98148895c754316e6eb4823d5cf033a0ee11d26e0cf67eeef80d3e Nov 04 03:26:23.196982 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:23.196951 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ftccw" event={"ID":"5a08f9b6-140f-4f6b-974a-dcce183d79c8","Type":"ContainerStarted","Data":"dd4b32b2607aa1b32f2ff4eef62fa6742b5e1a63251c254e292badc6fbaeb7b3"} Nov 04 03:26:23.197394 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:23.197004 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ftccw" event={"ID":"5a08f9b6-140f-4f6b-974a-dcce183d79c8","Type":"ContainerStarted","Data":"657009e88c98148895c754316e6eb4823d5cf033a0ee11d26e0cf67eeef80d3e"} Nov 04 03:26:23.317439 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:23.317381 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ftccw" podStartSLOduration=1.7321204959999998 podStartE2EDuration="2.317362005s" totalImagesPullingTime="585.241509ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:26:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:26:23.314541665 +0000 UTC m=+508.589172426" watchObservedRunningTime="2026-11-04 03:26:23.317362005 +0000 UTC m=+508.591992769" Nov 04 03:26:24.016055 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:24.016021 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ftccw" Nov 04 03:26:28.085112 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:28.085078 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmqzr" Nov 04 03:26:28.097644 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:28.097594 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmqzr" Nov 04 03:26:28.231256 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:28.231230 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmqzr" Nov 04 03:26:31.893850 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:31.893819 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq"] Nov 04 03:26:31.897521 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:31.897495 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq" Nov 04 03:26:31.982546 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:31.982512 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 04 03:26:32.001815 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:32.001794 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq"] Nov 04 03:26:32.016511 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:32.016440 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ftccw" Nov 04 03:26:32.028569 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:32.028548 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ftccw" Nov 04 03:26:32.034009 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:32.033988 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc86bf58-18cd-4838-a56b-7a88dde20533-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq\" (UID: \"dc86bf58-18cd-4838-a56b-7a88dde20533\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq" Nov 04 03:26:32.034093 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:32.034022 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc86bf58-18cd-4838-a56b-7a88dde20533-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq\" (UID: \"dc86bf58-18cd-4838-a56b-7a88dde20533\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq" Nov 04 03:26:32.034093 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:32.034053 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc86bf58-18cd-4838-a56b-7a88dde20533-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq\" (UID: \"dc86bf58-18cd-4838-a56b-7a88dde20533\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq" Nov 04 03:26:32.034202 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:32.034182 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc86bf58-18cd-4838-a56b-7a88dde20533-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq\" (UID: \"dc86bf58-18cd-4838-a56b-7a88dde20533\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq" Nov 04 03:26:32.034239 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:32.034217 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4svt\" (UniqueName: \"kubernetes.io/projected/dc86bf58-18cd-4838-a56b-7a88dde20533-kube-api-access-l4svt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq\" (UID: \"dc86bf58-18cd-4838-a56b-7a88dde20533\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq" Nov 04 03:26:32.034334 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:32.034317 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc86bf58-18cd-4838-a56b-7a88dde20533-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq\" (UID: \"dc86bf58-18cd-4838-a56b-7a88dde20533\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq" Nov 04 03:26:32.135518 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:32.135490 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc86bf58-18cd-4838-a56b-7a88dde20533-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq\" (UID: \"dc86bf58-18cd-4838-a56b-7a88dde20533\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq" Nov 04 03:26:32.135518 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:32.135524 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc86bf58-18cd-4838-a56b-7a88dde20533-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq\" (UID: \"dc86bf58-18cd-4838-a56b-7a88dde20533\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq" Nov 04 03:26:32.135746 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:32.135549 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc86bf58-18cd-4838-a56b-7a88dde20533-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq\" (UID: \"dc86bf58-18cd-4838-a56b-7a88dde20533\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq" Nov 04 03:26:32.135746 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:32.135642 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc86bf58-18cd-4838-a56b-7a88dde20533-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq\" (UID: \"dc86bf58-18cd-4838-a56b-7a88dde20533\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq" Nov 04 03:26:32.135746 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:32.135678 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4svt\" (UniqueName: \"kubernetes.io/projected/dc86bf58-18cd-4838-a56b-7a88dde20533-kube-api-access-l4svt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq\" (UID: \"dc86bf58-18cd-4838-a56b-7a88dde20533\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq" Nov 04 03:26:32.135903 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:32.135743 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc86bf58-18cd-4838-a56b-7a88dde20533-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq\" (UID: \"dc86bf58-18cd-4838-a56b-7a88dde20533\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq" Nov 04 03:26:32.136014 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:32.135990 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc86bf58-18cd-4838-a56b-7a88dde20533-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq\" (UID: \"dc86bf58-18cd-4838-a56b-7a88dde20533\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq" Nov 04 03:26:32.136114 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:32.136012 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc86bf58-18cd-4838-a56b-7a88dde20533-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq\" (UID: \"dc86bf58-18cd-4838-a56b-7a88dde20533\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq" Nov 04 03:26:32.136180 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:32.136157 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc86bf58-18cd-4838-a56b-7a88dde20533-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq\" (UID: \"dc86bf58-18cd-4838-a56b-7a88dde20533\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq" Nov 04 03:26:32.138033 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:32.138009 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc86bf58-18cd-4838-a56b-7a88dde20533-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq\" (UID: \"dc86bf58-18cd-4838-a56b-7a88dde20533\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq" Nov 04 03:26:32.138216 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:32.138197 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc86bf58-18cd-4838-a56b-7a88dde20533-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq\" (UID: \"dc86bf58-18cd-4838-a56b-7a88dde20533\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq" Nov 04 03:26:32.197220 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:32.197155 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4svt\" (UniqueName: \"kubernetes.io/projected/dc86bf58-18cd-4838-a56b-7a88dde20533-kube-api-access-l4svt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq\" (UID: \"dc86bf58-18cd-4838-a56b-7a88dde20533\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq" Nov 04 03:26:32.210053 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:32.210029 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq" Nov 04 03:26:32.258930 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:32.258897 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-ftccw" Nov 04 03:26:32.586928 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:32.586898 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq"] Nov 04 03:26:32.590057 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:26:32.590025 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc86bf58_18cd_4838_a56b_7a88dde20533.slice/crio-e3a24d0e14f65bcbd1a955a963fb308fb95b3f5e923a7bdc63e844b8363dd01e WatchSource:0}: Error finding container e3a24d0e14f65bcbd1a955a963fb308fb95b3f5e923a7bdc63e844b8363dd01e: Status 404 returned error can't find the container with id e3a24d0e14f65bcbd1a955a963fb308fb95b3f5e923a7bdc63e844b8363dd01e Nov 04 03:26:32.591913 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:32.591892 5953 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 03:26:33.252656 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:33.252595 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq" event={"ID":"dc86bf58-18cd-4838-a56b-7a88dde20533","Type":"ContainerStarted","Data":"e3a24d0e14f65bcbd1a955a963fb308fb95b3f5e923a7bdc63e844b8363dd01e"} Nov 04 03:26:34.259705 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:34.259667 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq" event={"ID":"dc86bf58-18cd-4838-a56b-7a88dde20533","Type":"ContainerStarted","Data":"05dbc3d9296a447639bf95eade6396b4555011bb6f83561f7a1c0e9f5f462587"} Nov 04 03:26:34.383790 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:34.383740 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq" podStartSLOduration=2.7768608649999997 podStartE2EDuration="3.383725857s" totalImagesPullingTime="606.864992ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:26:34.3823383 +0000 UTC m=+519.656969062" watchObservedRunningTime="2026-11-04 03:26:34.383725857 +0000 UTC m=+519.658356620" Nov 04 03:26:35.211051 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:35.211006 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq" Nov 04 03:26:38.734093 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:38.734052 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c"] Nov 04 03:26:38.738012 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:38.737991 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c" Nov 04 03:26:38.801828 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:38.801786 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d1425e2-49d3-462b-b7b5-85fd5e7adaf4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c\" (UID: \"3d1425e2-49d3-462b-b7b5-85fd5e7adaf4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c" Nov 04 03:26:38.802006 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:38.801906 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d1425e2-49d3-462b-b7b5-85fd5e7adaf4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c\" (UID: \"3d1425e2-49d3-462b-b7b5-85fd5e7adaf4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c" Nov 04 03:26:38.802006 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:38.801985 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d1425e2-49d3-462b-b7b5-85fd5e7adaf4-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c\" (UID: \"3d1425e2-49d3-462b-b7b5-85fd5e7adaf4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c" Nov 04 03:26:38.802097 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:38.802007 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d1425e2-49d3-462b-b7b5-85fd5e7adaf4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c\" (UID: \"3d1425e2-49d3-462b-b7b5-85fd5e7adaf4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c" Nov 04 03:26:38.802138 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:38.802121 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pwl9\" (UniqueName: \"kubernetes.io/projected/3d1425e2-49d3-462b-b7b5-85fd5e7adaf4-kube-api-access-6pwl9\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c\" (UID: \"3d1425e2-49d3-462b-b7b5-85fd5e7adaf4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c" Nov 04 03:26:38.802179 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:38.802145 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d1425e2-49d3-462b-b7b5-85fd5e7adaf4-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c\" (UID: \"3d1425e2-49d3-462b-b7b5-85fd5e7adaf4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c" Nov 04 03:26:38.840834 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:38.840799 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 04 03:26:38.842558 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:38.842526 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c"] Nov 04 03:26:38.903432 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:38.903390 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d1425e2-49d3-462b-b7b5-85fd5e7adaf4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c\" (UID: \"3d1425e2-49d3-462b-b7b5-85fd5e7adaf4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c" Nov 04 03:26:38.903678 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:38.903480 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d1425e2-49d3-462b-b7b5-85fd5e7adaf4-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c\" (UID: \"3d1425e2-49d3-462b-b7b5-85fd5e7adaf4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c" Nov 04 03:26:38.903678 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:38.903508 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d1425e2-49d3-462b-b7b5-85fd5e7adaf4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c\" (UID: \"3d1425e2-49d3-462b-b7b5-85fd5e7adaf4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c" Nov 04 03:26:38.903678 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:38.903588 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pwl9\" (UniqueName: \"kubernetes.io/projected/3d1425e2-49d3-462b-b7b5-85fd5e7adaf4-kube-api-access-6pwl9\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c\" (UID: \"3d1425e2-49d3-462b-b7b5-85fd5e7adaf4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c" Nov 04 03:26:38.903678 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:38.903615 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d1425e2-49d3-462b-b7b5-85fd5e7adaf4-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c\" (UID: \"3d1425e2-49d3-462b-b7b5-85fd5e7adaf4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c" Nov 04 03:26:38.903892 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:38.903691 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d1425e2-49d3-462b-b7b5-85fd5e7adaf4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c\" (UID: \"3d1425e2-49d3-462b-b7b5-85fd5e7adaf4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c" Nov 04 03:26:38.904793 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:38.904317 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d1425e2-49d3-462b-b7b5-85fd5e7adaf4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c\" (UID: \"3d1425e2-49d3-462b-b7b5-85fd5e7adaf4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c" Nov 04 03:26:38.904793 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:38.904484 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d1425e2-49d3-462b-b7b5-85fd5e7adaf4-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c\" (UID: \"3d1425e2-49d3-462b-b7b5-85fd5e7adaf4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c" Nov 04 03:26:38.905653 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:38.905602 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d1425e2-49d3-462b-b7b5-85fd5e7adaf4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c\" (UID: \"3d1425e2-49d3-462b-b7b5-85fd5e7adaf4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c" Nov 04 03:26:38.912232 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:38.907501 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d1425e2-49d3-462b-b7b5-85fd5e7adaf4-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c\" (UID: \"3d1425e2-49d3-462b-b7b5-85fd5e7adaf4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c" Nov 04 03:26:38.912232 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:38.907828 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d1425e2-49d3-462b-b7b5-85fd5e7adaf4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c\" (UID: \"3d1425e2-49d3-462b-b7b5-85fd5e7adaf4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c" Nov 04 03:26:38.967675 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:38.967639 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pwl9\" (UniqueName: \"kubernetes.io/projected/3d1425e2-49d3-462b-b7b5-85fd5e7adaf4-kube-api-access-6pwl9\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c\" (UID: \"3d1425e2-49d3-462b-b7b5-85fd5e7adaf4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c" Nov 04 03:26:39.050940 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:39.050903 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c" Nov 04 03:26:39.430017 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:39.429990 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c"] Nov 04 03:26:39.432757 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:26:39.432734 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d1425e2_49d3_462b_b7b5_85fd5e7adaf4.slice/crio-b0ebdd54a98ea1f19e447dbd665231df11637a34b659180c80652a06e93542c5 WatchSource:0}: Error finding container b0ebdd54a98ea1f19e447dbd665231df11637a34b659180c80652a06e93542c5: Status 404 returned error can't find the container with id b0ebdd54a98ea1f19e447dbd665231df11637a34b659180c80652a06e93542c5 Nov 04 03:26:40.289704 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:40.289665 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c" event={"ID":"3d1425e2-49d3-462b-b7b5-85fd5e7adaf4","Type":"ContainerStarted","Data":"c9135232110194185f4065a7425cee579f156ee19839392de0edb98fc11426d4"} Nov 04 03:26:40.290173 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:40.289723 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c" event={"ID":"3d1425e2-49d3-462b-b7b5-85fd5e7adaf4","Type":"ContainerStarted","Data":"b0ebdd54a98ea1f19e447dbd665231df11637a34b659180c80652a06e93542c5"} Nov 04 03:26:40.408450 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:40.408335 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c" podStartSLOduration=1.780696512 podStartE2EDuration="2.408318988s" totalImagesPullingTime="627.622476ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:26:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:26:40.407537737 +0000 UTC m=+525.682168496" watchObservedRunningTime="2026-11-04 03:26:40.408318988 +0000 UTC m=+525.682949750" Nov 04 03:26:41.051754 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:41.051710 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c" Nov 04 03:26:42.210464 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:42.210434 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq" Nov 04 03:26:42.223570 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:42.223537 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq" Nov 04 03:26:42.311334 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:42.311301 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq" Nov 04 03:26:46.386520 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:26:46.386412 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:26:48.297525 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.297491 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg"] Nov 04 03:26:48.301348 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.301328 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" Nov 04 03:26:48.347841 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.347800 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4"] Nov 04 03:26:48.351845 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.351820 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" Nov 04 03:26:48.354022 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.354000 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 04 03:26:48.368434 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.368404 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-cr7gn\"" Nov 04 03:26:48.401686 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.401609 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0122d333-f982-4664-bb39-62f707a44370-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdxg\" (UID: \"0122d333-f982-4664-bb39-62f707a44370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" Nov 04 03:26:48.401861 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.401741 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0122d333-f982-4664-bb39-62f707a44370-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdxg\" (UID: \"0122d333-f982-4664-bb39-62f707a44370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" Nov 04 03:26:48.401861 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.401814 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0122d333-f982-4664-bb39-62f707a44370-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdxg\" (UID: \"0122d333-f982-4664-bb39-62f707a44370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" Nov 04 03:26:48.401861 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.401851 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0122d333-f982-4664-bb39-62f707a44370-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdxg\" (UID: \"0122d333-f982-4664-bb39-62f707a44370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" Nov 04 03:26:48.401971 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.401921 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0122d333-f982-4664-bb39-62f707a44370-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdxg\" (UID: \"0122d333-f982-4664-bb39-62f707a44370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" Nov 04 03:26:48.401971 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.401949 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0122d333-f982-4664-bb39-62f707a44370-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdxg\" (UID: \"0122d333-f982-4664-bb39-62f707a44370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" Nov 04 03:26:48.402032 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.401972 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0122d333-f982-4664-bb39-62f707a44370-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdxg\" (UID: \"0122d333-f982-4664-bb39-62f707a44370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" Nov 04 03:26:48.402126 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.402103 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0122d333-f982-4664-bb39-62f707a44370-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdxg\" (UID: \"0122d333-f982-4664-bb39-62f707a44370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" Nov 04 03:26:48.402164 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.402149 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r25gs\" (UniqueName: \"kubernetes.io/projected/0122d333-f982-4664-bb39-62f707a44370-kube-api-access-r25gs\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdxg\" (UID: \"0122d333-f982-4664-bb39-62f707a44370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" Nov 04 03:26:48.405811 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.405776 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg"] Nov 04 03:26:48.456171 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.456136 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4"] Nov 04 03:26:48.503635 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.503592 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0122d333-f982-4664-bb39-62f707a44370-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdxg\" (UID: \"0122d333-f982-4664-bb39-62f707a44370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" Nov 04 03:26:48.503838 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.503652 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-vtsx4\" (UID: \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" Nov 04 03:26:48.503838 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.503689 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0122d333-f982-4664-bb39-62f707a44370-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdxg\" (UID: \"0122d333-f982-4664-bb39-62f707a44370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" Nov 04 03:26:48.503838 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.503719 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0122d333-f982-4664-bb39-62f707a44370-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdxg\" (UID: \"0122d333-f982-4664-bb39-62f707a44370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" Nov 04 03:26:48.503838 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.503753 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-vtsx4\" (UID: \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" Nov 04 03:26:48.504059 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.503847 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-vtsx4\" (UID: \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" Nov 04 03:26:48.504059 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.503891 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0122d333-f982-4664-bb39-62f707a44370-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdxg\" (UID: \"0122d333-f982-4664-bb39-62f707a44370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" Nov 04 03:26:48.504059 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.503923 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r25gs\" (UniqueName: \"kubernetes.io/projected/0122d333-f982-4664-bb39-62f707a44370-kube-api-access-r25gs\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdxg\" (UID: \"0122d333-f982-4664-bb39-62f707a44370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" Nov 04 03:26:48.504059 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.503966 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-vtsx4\" (UID: \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" Nov 04 03:26:48.504262 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.504138 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-vtsx4\" (UID: \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" Nov 04 03:26:48.504460 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.504436 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0122d333-f982-4664-bb39-62f707a44370-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdxg\" (UID: \"0122d333-f982-4664-bb39-62f707a44370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" Nov 04 03:26:48.504585 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.504485 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-vtsx4\" (UID: \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" Nov 04 03:26:48.504585 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.504495 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0122d333-f982-4664-bb39-62f707a44370-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdxg\" (UID: \"0122d333-f982-4664-bb39-62f707a44370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" Nov 04 03:26:48.504702 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.504655 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0122d333-f982-4664-bb39-62f707a44370-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdxg\" (UID: \"0122d333-f982-4664-bb39-62f707a44370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" Nov 04 03:26:48.504737 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.504706 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0122d333-f982-4664-bb39-62f707a44370-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdxg\" (UID: \"0122d333-f982-4664-bb39-62f707a44370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" Nov 04 03:26:48.504771 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.504759 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0122d333-f982-4664-bb39-62f707a44370-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdxg\" (UID: \"0122d333-f982-4664-bb39-62f707a44370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" Nov 04 03:26:48.504820 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.504800 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-vtsx4\" (UID: \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" Nov 04 03:26:48.505241 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.505202 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-vtsx4\" (UID: \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" Nov 04 03:26:48.505335 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.505255 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twswk\" (UniqueName: \"kubernetes.io/projected/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-kube-api-access-twswk\") pod \"maas-default-gateway-openshift-default-6dcf757899-vtsx4\" (UID: \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" Nov 04 03:26:48.505397 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.505344 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0122d333-f982-4664-bb39-62f707a44370-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdxg\" (UID: \"0122d333-f982-4664-bb39-62f707a44370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" Nov 04 03:26:48.505687 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.505668 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0122d333-f982-4664-bb39-62f707a44370-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdxg\" (UID: \"0122d333-f982-4664-bb39-62f707a44370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" Nov 04 03:26:48.505759 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.505706 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0122d333-f982-4664-bb39-62f707a44370-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdxg\" (UID: \"0122d333-f982-4664-bb39-62f707a44370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" Nov 04 03:26:48.506047 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.506027 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0122d333-f982-4664-bb39-62f707a44370-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdxg\" (UID: \"0122d333-f982-4664-bb39-62f707a44370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" Nov 04 03:26:48.506707 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.506686 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0122d333-f982-4664-bb39-62f707a44370-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdxg\" (UID: \"0122d333-f982-4664-bb39-62f707a44370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" Nov 04 03:26:48.506800 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.506758 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0122d333-f982-4664-bb39-62f707a44370-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdxg\" (UID: \"0122d333-f982-4664-bb39-62f707a44370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" Nov 04 03:26:48.566081 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.565999 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r25gs\" (UniqueName: \"kubernetes.io/projected/0122d333-f982-4664-bb39-62f707a44370-kube-api-access-r25gs\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdxg\" (UID: \"0122d333-f982-4664-bb39-62f707a44370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" Nov 04 03:26:48.566081 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.566001 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0122d333-f982-4664-bb39-62f707a44370-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdxg\" (UID: \"0122d333-f982-4664-bb39-62f707a44370\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" Nov 04 03:26:48.606686 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.606614 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-vtsx4\" (UID: \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" Nov 04 03:26:48.606893 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.606703 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-vtsx4\" (UID: \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" Nov 04 03:26:48.606893 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.606724 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-vtsx4\" (UID: \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" Nov 04 03:26:48.606893 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.606750 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-vtsx4\" (UID: \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" Nov 04 03:26:48.606893 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.606775 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-vtsx4\" (UID: \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" Nov 04 03:26:48.606893 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.606793 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-vtsx4\" (UID: \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" Nov 04 03:26:48.606893 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.606883 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-vtsx4\" (UID: \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" Nov 04 03:26:48.607251 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.606907 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-vtsx4\" (UID: \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" Nov 04 03:26:48.607251 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.606935 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twswk\" (UniqueName: \"kubernetes.io/projected/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-kube-api-access-twswk\") pod \"maas-default-gateway-openshift-default-6dcf757899-vtsx4\" (UID: \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" Nov 04 03:26:48.607366 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.607281 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-vtsx4\" (UID: \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" Nov 04 03:26:48.607366 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.607347 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-vtsx4\" (UID: \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" Nov 04 03:26:48.607485 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.607456 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-vtsx4\" (UID: \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" Nov 04 03:26:48.607538 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.607490 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-vtsx4\" (UID: \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" Nov 04 03:26:48.607610 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.607591 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-vtsx4\" (UID: \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" Nov 04 03:26:48.609221 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.609193 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-vtsx4\" (UID: \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" Nov 04 03:26:48.610297 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.610279 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-vtsx4\" (UID: \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" Nov 04 03:26:48.616280 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.616258 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" Nov 04 03:26:48.669785 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.669339 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-vtsx4\" (UID: \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" Nov 04 03:26:48.670929 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.670596 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twswk\" (UniqueName: \"kubernetes.io/projected/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-kube-api-access-twswk\") pod \"maas-default-gateway-openshift-default-6dcf757899-vtsx4\" (UID: \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" Nov 04 03:26:48.965424 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:48.965331 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" Nov 04 03:26:49.044486 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:49.044446 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg"] Nov 04 03:26:49.048212 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:26:49.048174 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0122d333_f982_4664_bb39_62f707a44370.slice/crio-fd4eb46367cb6c42de06f2fc32bde782023533a14c6df70b5a2eb5fe85911882 WatchSource:0}: Error finding container fd4eb46367cb6c42de06f2fc32bde782023533a14c6df70b5a2eb5fe85911882: Status 404 returned error can't find the container with id fd4eb46367cb6c42de06f2fc32bde782023533a14c6df70b5a2eb5fe85911882 Nov 04 03:26:49.051311 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:49.051027 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c" Nov 04 03:26:49.063937 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:49.063908 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c" Nov 04 03:26:49.335441 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:49.335396 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" event={"ID":"0122d333-f982-4664-bb39-62f707a44370","Type":"ContainerStarted","Data":"fd4eb46367cb6c42de06f2fc32bde782023533a14c6df70b5a2eb5fe85911882"} Nov 04 03:26:49.346527 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:49.346483 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4"] Nov 04 03:26:49.347801 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:49.347779 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c" Nov 04 03:26:49.351947 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:26:49.351918 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod162aa7e4_1372_460b_b6eb_2d12fbb5ea51.slice/crio-9c827252ddc0a707a102846f4ca61e49869b73afe375a131bef02d48fad26517 WatchSource:0}: Error finding container 9c827252ddc0a707a102846f4ca61e49869b73afe375a131bef02d48fad26517: Status 404 returned error can't find the container with id 9c827252ddc0a707a102846f4ca61e49869b73afe375a131bef02d48fad26517 Nov 04 03:26:50.341614 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:50.341567 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" event={"ID":"162aa7e4-1372-460b-b6eb-2d12fbb5ea51","Type":"ContainerStarted","Data":"9c827252ddc0a707a102846f4ca61e49869b73afe375a131bef02d48fad26517"} Nov 04 03:26:53.190732 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:53.190684 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:26:53.191068 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:53.190786 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:26:53.191068 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:53.190830 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:26:53.286989 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:53.286956 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:26:53.287083 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:53.287032 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:26:53.287083 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:53.287061 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:26:53.365009 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:53.364973 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" event={"ID":"0122d333-f982-4664-bb39-62f707a44370","Type":"ContainerStarted","Data":"bbb2bdad1085894658fb18e0a23ae75abadbaee44573bd537abd35e8a201a399"} Nov 04 03:26:53.505353 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:53.505290 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" podStartSLOduration=1.366094089 podStartE2EDuration="5.505271855s" totalImagesPullingTime="4.139177766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:26:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:26:53.503100993 +0000 UTC m=+538.777731765" watchObservedRunningTime="2026-11-04 03:26:53.505271855 +0000 UTC m=+538.779902622" Nov 04 03:26:53.616970 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:53.616940 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" Nov 04 03:26:54.371939 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:54.371896 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" event={"ID":"162aa7e4-1372-460b-b6eb-2d12fbb5ea51","Type":"ContainerStarted","Data":"a214581bfc771d5977c465decc892e6281e5011dc1f2c1112a04ba6a7cb094ef"} Nov 04 03:26:54.624706 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:54.622984 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" podStartSLOduration=2.690822533 podStartE2EDuration="6.622963815s" totalImagesPullingTime="3.932141282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:26:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:26:54.557161752 +0000 UTC m=+539.831792516" watchObservedRunningTime="2026-11-04 03:26:54.622963815 +0000 UTC m=+539.897594583" Nov 04 03:26:54.624706 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:54.624464 5953 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-skdxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 04 03:26:54.624706 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:54.624591 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" podUID="0122d333-f982-4664-bb39-62f707a44370" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 04 03:26:54.966118 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:54.966087 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" Nov 04 03:26:54.967669 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:54.967645 5953 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vtsx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 04 03:26:54.967841 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:54.967820 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" podUID="162aa7e4-1372-460b-b6eb-2d12fbb5ea51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 04 03:26:55.622647 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:55.620048 5953 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-skdxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 04 03:26:55.622647 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:55.620115 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" podUID="0122d333-f982-4664-bb39-62f707a44370" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 04 03:26:55.966523 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:55.966420 5953 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vtsx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 04 03:26:55.966728 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:55.966501 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" podUID="162aa7e4-1372-460b-b6eb-2d12fbb5ea51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 04 03:26:56.622661 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:56.621037 5953 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-skdxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 04 03:26:56.622661 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:56.621105 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" podUID="0122d333-f982-4664-bb39-62f707a44370" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 04 03:26:56.966301 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:56.966207 5953 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vtsx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 04 03:26:56.966301 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:56.966276 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" podUID="162aa7e4-1372-460b-b6eb-2d12fbb5ea51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 04 03:26:57.625645 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:57.624101 5953 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-skdxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 04 03:26:57.625645 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:57.624185 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" podUID="0122d333-f982-4664-bb39-62f707a44370" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 04 03:26:57.966614 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:57.966528 5953 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vtsx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 04 03:26:57.966614 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:57.966589 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" podUID="162aa7e4-1372-460b-b6eb-2d12fbb5ea51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 04 03:26:58.618148 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:58.618110 5953 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-skdxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 04 03:26:58.618452 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:58.618425 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" podUID="0122d333-f982-4664-bb39-62f707a44370" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 04 03:26:58.966130 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:58.966041 5953 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vtsx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 04 03:26:58.966130 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:58.966098 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" podUID="162aa7e4-1372-460b-b6eb-2d12fbb5ea51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 04 03:26:59.617049 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:59.617015 5953 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-skdxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 04 03:26:59.617237 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:59.617080 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" podUID="0122d333-f982-4664-bb39-62f707a44370" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 04 03:26:59.966080 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:59.965980 5953 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vtsx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 04 03:26:59.966080 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:26:59.966042 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" podUID="162aa7e4-1372-460b-b6eb-2d12fbb5ea51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 04 03:27:00.616706 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:27:00.616664 5953 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-skdxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 04 03:27:00.616893 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:27:00.616728 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" podUID="0122d333-f982-4664-bb39-62f707a44370" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 04 03:27:00.966169 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:27:00.966064 5953 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vtsx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 04 03:27:00.966169 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:27:00.966120 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" podUID="162aa7e4-1372-460b-b6eb-2d12fbb5ea51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 04 03:27:02.049131 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:27:02.049097 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" Nov 04 03:27:02.050029 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:27:02.050001 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" Nov 04 03:27:02.051125 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:27:02.051099 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" Nov 04 03:27:02.083543 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:27:02.083513 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" Nov 04 03:27:02.084156 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:27:02.084136 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" Nov 04 03:27:02.084882 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:27:02.084864 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" Nov 04 03:27:16.385413 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:27:16.385325 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:27:21.419910 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:27:21.419869 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-874888cd5-l4bhw"] Nov 04 03:27:21.424130 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:27:21.424107 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-874888cd5-l4bhw" Nov 04 03:27:21.480774 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:27:21.480743 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-874888cd5-l4bhw"] Nov 04 03:27:21.529236 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:27:21.529201 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 04 03:27:21.529236 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:27:21.529201 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9ktvr\"" Nov 04 03:27:21.577420 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:27:21.577371 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2kpj\" (UniqueName: \"kubernetes.io/projected/a67b383f-8c42-4aca-a45c-2666a066a80e-kube-api-access-q2kpj\") pod \"authorino-874888cd5-l4bhw\" (UID: \"a67b383f-8c42-4aca-a45c-2666a066a80e\") " pod="kuadrant-system/authorino-874888cd5-l4bhw" Nov 04 03:27:21.577658 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:27:21.577564 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a67b383f-8c42-4aca-a45c-2666a066a80e-tls-cert\") pod \"authorino-874888cd5-l4bhw\" (UID: \"a67b383f-8c42-4aca-a45c-2666a066a80e\") " pod="kuadrant-system/authorino-874888cd5-l4bhw" Nov 04 03:27:21.679180 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:27:21.679075 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a67b383f-8c42-4aca-a45c-2666a066a80e-tls-cert\") pod \"authorino-874888cd5-l4bhw\" (UID: \"a67b383f-8c42-4aca-a45c-2666a066a80e\") " pod="kuadrant-system/authorino-874888cd5-l4bhw" Nov 04 03:27:21.679341 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:27:21.679190 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2kpj\" (UniqueName: \"kubernetes.io/projected/a67b383f-8c42-4aca-a45c-2666a066a80e-kube-api-access-q2kpj\") pod \"authorino-874888cd5-l4bhw\" (UID: \"a67b383f-8c42-4aca-a45c-2666a066a80e\") " pod="kuadrant-system/authorino-874888cd5-l4bhw" Nov 04 03:27:21.681804 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:27:21.681780 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a67b383f-8c42-4aca-a45c-2666a066a80e-tls-cert\") pod \"authorino-874888cd5-l4bhw\" (UID: \"a67b383f-8c42-4aca-a45c-2666a066a80e\") " pod="kuadrant-system/authorino-874888cd5-l4bhw" Nov 04 03:27:21.741247 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:27:21.741219 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2kpj\" (UniqueName: \"kubernetes.io/projected/a67b383f-8c42-4aca-a45c-2666a066a80e-kube-api-access-q2kpj\") pod \"authorino-874888cd5-l4bhw\" (UID: \"a67b383f-8c42-4aca-a45c-2666a066a80e\") " pod="kuadrant-system/authorino-874888cd5-l4bhw" Nov 04 03:27:22.036456 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:27:22.036417 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-874888cd5-l4bhw" Nov 04 03:27:22.413027 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:27:22.413001 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-874888cd5-l4bhw"] Nov 04 03:27:22.415874 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:27:22.415851 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda67b383f_8c42_4aca_a45c_2666a066a80e.slice/crio-01fdec4c8d7bbb26ad85cb1eb3e4456086bd18c89a7879b826c792ea36ee67ee WatchSource:0}: Error finding container 01fdec4c8d7bbb26ad85cb1eb3e4456086bd18c89a7879b826c792ea36ee67ee: Status 404 returned error can't find the container with id 01fdec4c8d7bbb26ad85cb1eb3e4456086bd18c89a7879b826c792ea36ee67ee Nov 04 03:27:22.534309 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:27:22.534276 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-874888cd5-l4bhw" event={"ID":"a67b383f-8c42-4aca-a45c-2666a066a80e","Type":"ContainerStarted","Data":"01fdec4c8d7bbb26ad85cb1eb3e4456086bd18c89a7879b826c792ea36ee67ee"} Nov 04 03:27:24.549742 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:27:24.549696 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-874888cd5-l4bhw" event={"ID":"a67b383f-8c42-4aca-a45c-2666a066a80e","Type":"ContainerStarted","Data":"f821aa3111cbb9f3f7a981ef988005f56a33b87f84cfd8d51142f5b80cfbfab8"} Nov 04 03:27:24.666974 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:27:24.666919 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-874888cd5-l4bhw" podStartSLOduration=2.138998953 podStartE2EDuration="3.666901581s" totalImagesPullingTime="1.527902628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:27:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:27:24.665788881 +0000 UTC m=+569.940419647" watchObservedRunningTime="2026-11-04 03:27:24.666901581 +0000 UTC m=+569.941532342" Nov 04 03:27:46.385780 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:27:46.385736 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:28:16.384553 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:28:16.384515 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:28:46.384206 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:28:46.384167 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:28:54.493799 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:54.493759 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f"] Nov 04 03:28:54.497962 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:54.497939 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" Nov 04 03:28:54.557530 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:54.557503 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-c5bec0-openshift-default-dockercfg-fkbcb\"" Nov 04 03:28:54.567785 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:54.567749 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdfcbe1f-229d-4c24-9e85-01e59dac657a-istio-podinfo\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f\" (UID: \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" Nov 04 03:28:54.567785 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:54.567796 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdfcbe1f-229d-4c24-9e85-01e59dac657a-istio-token\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f\" (UID: \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" Nov 04 03:28:54.568028 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:54.567885 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdfcbe1f-229d-4c24-9e85-01e59dac657a-credential-socket\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f\" (UID: \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" Nov 04 03:28:54.568028 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:54.567943 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdfcbe1f-229d-4c24-9e85-01e59dac657a-istiod-ca-cert\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f\" (UID: \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" Nov 04 03:28:54.568028 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:54.567974 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgb8x\" (UniqueName: \"kubernetes.io/projected/cdfcbe1f-229d-4c24-9e85-01e59dac657a-kube-api-access-dgb8x\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f\" (UID: \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" Nov 04 03:28:54.568028 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:54.567998 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdfcbe1f-229d-4c24-9e85-01e59dac657a-istio-envoy\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f\" (UID: \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" Nov 04 03:28:54.568028 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:54.568021 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdfcbe1f-229d-4c24-9e85-01e59dac657a-workload-certs\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f\" (UID: \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" Nov 04 03:28:54.568205 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:54.568053 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdfcbe1f-229d-4c24-9e85-01e59dac657a-istio-data\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f\" (UID: \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" Nov 04 03:28:54.568205 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:54.568099 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdfcbe1f-229d-4c24-9e85-01e59dac657a-workload-socket\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f\" (UID: \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" Nov 04 03:28:54.604177 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:54.604147 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f"] Nov 04 03:28:54.669407 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:54.669372 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdfcbe1f-229d-4c24-9e85-01e59dac657a-istio-podinfo\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f\" (UID: \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" Nov 04 03:28:54.669407 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:54.669411 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdfcbe1f-229d-4c24-9e85-01e59dac657a-istio-token\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f\" (UID: \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" Nov 04 03:28:54.669739 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:54.669466 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdfcbe1f-229d-4c24-9e85-01e59dac657a-credential-socket\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f\" (UID: \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" Nov 04 03:28:54.669739 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:54.669498 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdfcbe1f-229d-4c24-9e85-01e59dac657a-istiod-ca-cert\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f\" (UID: \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" Nov 04 03:28:54.669739 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:54.669522 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgb8x\" (UniqueName: \"kubernetes.io/projected/cdfcbe1f-229d-4c24-9e85-01e59dac657a-kube-api-access-dgb8x\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f\" (UID: \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" Nov 04 03:28:54.669739 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:54.669551 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdfcbe1f-229d-4c24-9e85-01e59dac657a-istio-envoy\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f\" (UID: \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" Nov 04 03:28:54.669739 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:54.669589 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdfcbe1f-229d-4c24-9e85-01e59dac657a-workload-certs\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f\" (UID: \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" Nov 04 03:28:54.669739 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:54.669658 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdfcbe1f-229d-4c24-9e85-01e59dac657a-istio-data\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f\" (UID: \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" Nov 04 03:28:54.669739 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:54.669705 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdfcbe1f-229d-4c24-9e85-01e59dac657a-workload-socket\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f\" (UID: \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" Nov 04 03:28:54.670146 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:54.670125 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdfcbe1f-229d-4c24-9e85-01e59dac657a-credential-socket\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f\" (UID: \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" Nov 04 03:28:54.670233 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:54.670179 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdfcbe1f-229d-4c24-9e85-01e59dac657a-istio-data\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f\" (UID: \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" Nov 04 03:28:54.670286 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:54.670259 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdfcbe1f-229d-4c24-9e85-01e59dac657a-workload-certs\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f\" (UID: \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" Nov 04 03:28:54.670413 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:54.670395 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdfcbe1f-229d-4c24-9e85-01e59dac657a-istiod-ca-cert\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f\" (UID: \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" Nov 04 03:28:54.670556 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:54.670539 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdfcbe1f-229d-4c24-9e85-01e59dac657a-workload-socket\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f\" (UID: \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" Nov 04 03:28:54.672161 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:54.672137 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdfcbe1f-229d-4c24-9e85-01e59dac657a-istio-envoy\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f\" (UID: \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" Nov 04 03:28:54.672246 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:54.672228 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdfcbe1f-229d-4c24-9e85-01e59dac657a-istio-podinfo\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f\" (UID: \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" Nov 04 03:28:54.731515 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:54.731479 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdfcbe1f-229d-4c24-9e85-01e59dac657a-istio-token\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f\" (UID: \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" Nov 04 03:28:54.731684 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:54.731530 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgb8x\" (UniqueName: \"kubernetes.io/projected/cdfcbe1f-229d-4c24-9e85-01e59dac657a-kube-api-access-dgb8x\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f\" (UID: \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" Nov 04 03:28:54.812107 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:54.812077 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" Nov 04 03:28:55.243513 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:55.243290 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f"] Nov 04 03:28:55.246438 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:28:55.246411 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdfcbe1f_229d_4c24_9e85_01e59dac657a.slice/crio-dbcb32d2fe164b6bd5087b67fef62276af624ae709e920bf1671ff2a8118e64b WatchSource:0}: Error finding container dbcb32d2fe164b6bd5087b67fef62276af624ae709e920bf1671ff2a8118e64b: Status 404 returned error can't find the container with id dbcb32d2fe164b6bd5087b67fef62276af624ae709e920bf1671ff2a8118e64b Nov 04 03:28:55.248864 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:55.248823 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:28:55.248973 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:55.248901 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:28:55.249036 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:55.248969 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:28:56.015442 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:56.015403 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" event={"ID":"cdfcbe1f-229d-4c24-9e85-01e59dac657a","Type":"ContainerStarted","Data":"15061a35835d7a806d91781392f465eb88d2d8dad663d6687aa49a7db223b4b7"} Nov 04 03:28:56.015848 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:56.015465 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" event={"ID":"cdfcbe1f-229d-4c24-9e85-01e59dac657a","Type":"ContainerStarted","Data":"dbcb32d2fe164b6bd5087b67fef62276af624ae709e920bf1671ff2a8118e64b"} Nov 04 03:28:56.237593 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:56.237542 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" podStartSLOduration=2.237528599 podStartE2EDuration="2.237528599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:28:56.186684629 +0000 UTC m=+661.461315392" watchObservedRunningTime="2026-11-04 03:28:56.237528599 +0000 UTC m=+661.512159360" Nov 04 03:28:56.812938 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:56.812892 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" Nov 04 03:28:56.818312 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:56.818283 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" Nov 04 03:28:57.021549 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:57.021514 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" Nov 04 03:28:57.022451 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:28:57.022431 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" Nov 04 03:29:01.423636 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.423595 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-10f834-6d5d8b948c-fclqf"] Nov 04 03:29:01.427768 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.427745 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-10f834-6d5d8b948c-fclqf" Nov 04 03:29:01.474053 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.474022 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8"] Nov 04 03:29:01.477937 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.477920 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" Nov 04 03:29:01.484184 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.484161 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-10f834-dockercfg-9t5c9\"" Nov 04 03:29:01.494000 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.493976 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-10f834\"" Nov 04 03:29:01.496932 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.496904 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-10f834-6d5d8b948c-fclqf"] Nov 04 03:29:01.533354 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.533321 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8"] Nov 04 03:29:01.545130 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.545098 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9d98901-eb52-4bd0-a031-640623bd4272-workload-socket\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8\" (UID: \"e9d98901-eb52-4bd0-a031-640623bd4272\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" Nov 04 03:29:01.545297 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.545138 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9d98901-eb52-4bd0-a031-640623bd4272-credential-socket\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8\" (UID: \"e9d98901-eb52-4bd0-a031-640623bd4272\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" Nov 04 03:29:01.545297 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.545167 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9d98901-eb52-4bd0-a031-640623bd4272-istiod-ca-cert\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8\" (UID: \"e9d98901-eb52-4bd0-a031-640623bd4272\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" Nov 04 03:29:01.545445 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.545292 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9d98901-eb52-4bd0-a031-640623bd4272-istio-podinfo\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8\" (UID: \"e9d98901-eb52-4bd0-a031-640623bd4272\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" Nov 04 03:29:01.545445 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.545330 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9d98901-eb52-4bd0-a031-640623bd4272-workload-certs\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8\" (UID: \"e9d98901-eb52-4bd0-a031-640623bd4272\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" Nov 04 03:29:01.545445 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.545361 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9d98901-eb52-4bd0-a031-640623bd4272-istio-token\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8\" (UID: \"e9d98901-eb52-4bd0-a031-640623bd4272\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" Nov 04 03:29:01.545445 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.545432 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9d98901-eb52-4bd0-a031-640623bd4272-istio-envoy\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8\" (UID: \"e9d98901-eb52-4bd0-a031-640623bd4272\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" Nov 04 03:29:01.545770 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.545488 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pp95\" (UniqueName: \"kubernetes.io/projected/e9d98901-eb52-4bd0-a031-640623bd4272-kube-api-access-7pp95\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8\" (UID: \"e9d98901-eb52-4bd0-a031-640623bd4272\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" Nov 04 03:29:01.545770 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.545518 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zddk\" (UniqueName: \"kubernetes.io/projected/9f801ad2-3210-4369-beac-73f789ae5559-kube-api-access-4zddk\") pod \"payload-pre-processing-e2e-worker-w4-10f834-6d5d8b948c-fclqf\" (UID: \"9f801ad2-3210-4369-beac-73f789ae5559\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-10f834-6d5d8b948c-fclqf" Nov 04 03:29:01.545770 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.545585 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f801ad2-3210-4369-beac-73f789ae5559-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-10f834-6d5d8b948c-fclqf\" (UID: \"9f801ad2-3210-4369-beac-73f789ae5559\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-10f834-6d5d8b948c-fclqf" Nov 04 03:29:01.546461 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.546442 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-90051a-openshift-default-dockercfg-88sl4\"" Nov 04 03:29:01.547005 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.546983 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9d98901-eb52-4bd0-a031-640623bd4272-istio-data\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8\" (UID: \"e9d98901-eb52-4bd0-a031-640623bd4272\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" Nov 04 03:29:01.647977 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.647946 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9d98901-eb52-4bd0-a031-640623bd4272-istio-data\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8\" (UID: \"e9d98901-eb52-4bd0-a031-640623bd4272\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" Nov 04 03:29:01.648173 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.648006 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9d98901-eb52-4bd0-a031-640623bd4272-workload-socket\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8\" (UID: \"e9d98901-eb52-4bd0-a031-640623bd4272\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" Nov 04 03:29:01.648173 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.648031 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9d98901-eb52-4bd0-a031-640623bd4272-credential-socket\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8\" (UID: \"e9d98901-eb52-4bd0-a031-640623bd4272\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" Nov 04 03:29:01.648173 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.648056 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9d98901-eb52-4bd0-a031-640623bd4272-istiod-ca-cert\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8\" (UID: \"e9d98901-eb52-4bd0-a031-640623bd4272\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" Nov 04 03:29:01.648173 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.648094 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9d98901-eb52-4bd0-a031-640623bd4272-istio-podinfo\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8\" (UID: \"e9d98901-eb52-4bd0-a031-640623bd4272\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" Nov 04 03:29:01.648173 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.648120 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9d98901-eb52-4bd0-a031-640623bd4272-workload-certs\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8\" (UID: \"e9d98901-eb52-4bd0-a031-640623bd4272\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" Nov 04 03:29:01.648173 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.648143 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9d98901-eb52-4bd0-a031-640623bd4272-istio-token\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8\" (UID: \"e9d98901-eb52-4bd0-a031-640623bd4272\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" Nov 04 03:29:01.648173 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.648166 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9d98901-eb52-4bd0-a031-640623bd4272-istio-envoy\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8\" (UID: \"e9d98901-eb52-4bd0-a031-640623bd4272\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" Nov 04 03:29:01.648534 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.648220 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pp95\" (UniqueName: \"kubernetes.io/projected/e9d98901-eb52-4bd0-a031-640623bd4272-kube-api-access-7pp95\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8\" (UID: \"e9d98901-eb52-4bd0-a031-640623bd4272\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" Nov 04 03:29:01.648534 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.648251 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zddk\" (UniqueName: \"kubernetes.io/projected/9f801ad2-3210-4369-beac-73f789ae5559-kube-api-access-4zddk\") pod \"payload-pre-processing-e2e-worker-w4-10f834-6d5d8b948c-fclqf\" (UID: \"9f801ad2-3210-4369-beac-73f789ae5559\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-10f834-6d5d8b948c-fclqf" Nov 04 03:29:01.648534 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.648281 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f801ad2-3210-4369-beac-73f789ae5559-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-10f834-6d5d8b948c-fclqf\" (UID: \"9f801ad2-3210-4369-beac-73f789ae5559\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-10f834-6d5d8b948c-fclqf" Nov 04 03:29:01.648534 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.648434 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9d98901-eb52-4bd0-a031-640623bd4272-istio-data\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8\" (UID: \"e9d98901-eb52-4bd0-a031-640623bd4272\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" Nov 04 03:29:01.648534 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.648456 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9d98901-eb52-4bd0-a031-640623bd4272-workload-socket\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8\" (UID: \"e9d98901-eb52-4bd0-a031-640623bd4272\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" Nov 04 03:29:01.648821 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.648536 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9d98901-eb52-4bd0-a031-640623bd4272-credential-socket\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8\" (UID: \"e9d98901-eb52-4bd0-a031-640623bd4272\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" Nov 04 03:29:01.648821 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.648572 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9d98901-eb52-4bd0-a031-640623bd4272-workload-certs\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8\" (UID: \"e9d98901-eb52-4bd0-a031-640623bd4272\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" Nov 04 03:29:01.648973 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.648951 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9d98901-eb52-4bd0-a031-640623bd4272-istiod-ca-cert\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8\" (UID: \"e9d98901-eb52-4bd0-a031-640623bd4272\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" Nov 04 03:29:01.649072 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.649051 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f801ad2-3210-4369-beac-73f789ae5559-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-10f834-6d5d8b948c-fclqf\" (UID: \"9f801ad2-3210-4369-beac-73f789ae5559\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-10f834-6d5d8b948c-fclqf" Nov 04 03:29:01.651328 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.651301 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9d98901-eb52-4bd0-a031-640623bd4272-istio-envoy\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8\" (UID: \"e9d98901-eb52-4bd0-a031-640623bd4272\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" Nov 04 03:29:01.651538 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.651520 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9d98901-eb52-4bd0-a031-640623bd4272-istio-podinfo\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8\" (UID: \"e9d98901-eb52-4bd0-a031-640623bd4272\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" Nov 04 03:29:01.714906 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.714830 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9d98901-eb52-4bd0-a031-640623bd4272-istio-token\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8\" (UID: \"e9d98901-eb52-4bd0-a031-640623bd4272\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" Nov 04 03:29:01.714906 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.714878 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zddk\" (UniqueName: \"kubernetes.io/projected/9f801ad2-3210-4369-beac-73f789ae5559-kube-api-access-4zddk\") pod \"payload-pre-processing-e2e-worker-w4-10f834-6d5d8b948c-fclqf\" (UID: \"9f801ad2-3210-4369-beac-73f789ae5559\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-10f834-6d5d8b948c-fclqf" Nov 04 03:29:01.715117 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.714947 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pp95\" (UniqueName: \"kubernetes.io/projected/e9d98901-eb52-4bd0-a031-640623bd4272-kube-api-access-7pp95\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8\" (UID: \"e9d98901-eb52-4bd0-a031-640623bd4272\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" Nov 04 03:29:01.741850 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.741820 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-10f834-6d5d8b948c-fclqf" Nov 04 03:29:01.792587 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:01.792552 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" Nov 04 03:29:02.123543 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:02.123516 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-10f834-6d5d8b948c-fclqf"] Nov 04 03:29:02.126251 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:29:02.126219 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f801ad2_3210_4369_beac_73f789ae5559.slice/crio-501bce7afb4513f3186502c3fad6edb4fe41d5a528645565b61df966bc9d4a47 WatchSource:0}: Error finding container 501bce7afb4513f3186502c3fad6edb4fe41d5a528645565b61df966bc9d4a47: Status 404 returned error can't find the container with id 501bce7afb4513f3186502c3fad6edb4fe41d5a528645565b61df966bc9d4a47 Nov 04 03:29:02.190881 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:02.190847 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8"] Nov 04 03:29:02.193041 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:29:02.193016 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9d98901_eb52_4bd0_a031_640623bd4272.slice/crio-8c7263742c8eebe1542edf5f17c0d8d3891ce97419bae31b7b268a34898ef5f1 WatchSource:0}: Error finding container 8c7263742c8eebe1542edf5f17c0d8d3891ce97419bae31b7b268a34898ef5f1: Status 404 returned error can't find the container with id 8c7263742c8eebe1542edf5f17c0d8d3891ce97419bae31b7b268a34898ef5f1 Nov 04 03:29:02.195858 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:02.195824 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:29:02.195956 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:02.195897 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:29:02.195956 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:02.195926 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:29:03.050554 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:03.050511 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" event={"ID":"e9d98901-eb52-4bd0-a031-640623bd4272","Type":"ContainerStarted","Data":"9494120f489a4d24f206df071b044ca6655db40dd6f4ac65f423685e8added78"} Nov 04 03:29:03.051011 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:03.050576 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" event={"ID":"e9d98901-eb52-4bd0-a031-640623bd4272","Type":"ContainerStarted","Data":"8c7263742c8eebe1542edf5f17c0d8d3891ce97419bae31b7b268a34898ef5f1"} Nov 04 03:29:03.052381 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:03.052357 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-10f834-6d5d8b948c-fclqf" event={"ID":"9f801ad2-3210-4369-beac-73f789ae5559","Type":"ContainerStarted","Data":"aa9f9c00738e14caf1f261924132f82fe5042be5cce1820ebc055c041005ad54"} Nov 04 03:29:03.052497 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:03.052399 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-10f834-6d5d8b948c-fclqf" event={"ID":"9f801ad2-3210-4369-beac-73f789ae5559","Type":"ContainerStarted","Data":"501bce7afb4513f3186502c3fad6edb4fe41d5a528645565b61df966bc9d4a47"} Nov 04 03:29:03.052497 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:03.052459 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-10f834-6d5d8b948c-fclqf" Nov 04 03:29:03.223722 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:03.223661 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" podStartSLOduration=2.223642058 podStartE2EDuration="2.223642058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:29:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:29:03.221827086 +0000 UTC m=+668.496457861" watchObservedRunningTime="2026-11-04 03:29:03.223642058 +0000 UTC m=+668.498272822" Nov 04 03:29:03.344047 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:03.343908 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-10f834-6d5d8b948c-fclqf" podStartSLOduration=2.343885449 podStartE2EDuration="2.343885449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:29:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:29:03.338882891 +0000 UTC m=+668.613513648" watchObservedRunningTime="2026-11-04 03:29:03.343885449 +0000 UTC m=+668.618516211" Nov 04 03:29:03.793050 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:03.793009 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" Nov 04 03:29:03.798479 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:03.798451 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" Nov 04 03:29:04.060807 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:04.060728 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm"] Nov 04 03:29:04.064598 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:04.064577 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" Nov 04 03:29:04.064751 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:04.064644 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" Nov 04 03:29:04.064751 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:04.064688 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm" Nov 04 03:29:04.118493 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:04.118329 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:29:04.118493 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:04.118409 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-10f834-serving-cert\"" Nov 04 03:29:04.118493 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:04.118477 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:29:04.119009 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:04.118646 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 04 03:29:04.119278 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:04.119257 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-m6mdd\"" Nov 04 03:29:04.121388 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:04.121365 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm"] Nov 04 03:29:04.174451 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:04.174420 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d72a213b-6993-4966-9527-2bb12da6bc8c-metrics-certs\") pod \"maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm\" (UID: \"d72a213b-6993-4966-9527-2bb12da6bc8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm" Nov 04 03:29:04.174665 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:04.174646 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg6g2\" (UniqueName: \"kubernetes.io/projected/d72a213b-6993-4966-9527-2bb12da6bc8c-kube-api-access-cg6g2\") pod \"maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm\" (UID: \"d72a213b-6993-4966-9527-2bb12da6bc8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm" Nov 04 03:29:04.174732 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:04.174694 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d72a213b-6993-4966-9527-2bb12da6bc8c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm\" (UID: \"d72a213b-6993-4966-9527-2bb12da6bc8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm" Nov 04 03:29:04.174899 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:04.174874 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d72a213b-6993-4966-9527-2bb12da6bc8c-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm\" (UID: \"d72a213b-6993-4966-9527-2bb12da6bc8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm" Nov 04 03:29:04.275631 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:04.275583 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg6g2\" (UniqueName: \"kubernetes.io/projected/d72a213b-6993-4966-9527-2bb12da6bc8c-kube-api-access-cg6g2\") pod \"maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm\" (UID: \"d72a213b-6993-4966-9527-2bb12da6bc8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm" Nov 04 03:29:04.275827 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:04.275648 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d72a213b-6993-4966-9527-2bb12da6bc8c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm\" (UID: \"d72a213b-6993-4966-9527-2bb12da6bc8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm" Nov 04 03:29:04.275827 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:04.275725 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d72a213b-6993-4966-9527-2bb12da6bc8c-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm\" (UID: \"d72a213b-6993-4966-9527-2bb12da6bc8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm" Nov 04 03:29:04.275827 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:29:04.275801 5953 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-10f834-serving-cert: secret "maas-api-e2e-worker-w4-10f834-serving-cert" not found Nov 04 03:29:04.275827 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:04.275816 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d72a213b-6993-4966-9527-2bb12da6bc8c-metrics-certs\") pod \"maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm\" (UID: \"d72a213b-6993-4966-9527-2bb12da6bc8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm" Nov 04 03:29:04.276052 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:29:04.275871 5953 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d72a213b-6993-4966-9527-2bb12da6bc8c-maas-api-tls podName:d72a213b-6993-4966-9527-2bb12da6bc8c nodeName:}" failed. No retries permitted until 2026-11-04 03:29:04.77585342 +0000 UTC m=+670.050484174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d72a213b-6993-4966-9527-2bb12da6bc8c-maas-api-tls") pod "maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm" (UID: "d72a213b-6993-4966-9527-2bb12da6bc8c") : secret "maas-api-e2e-worker-w4-10f834-serving-cert" not found Nov 04 03:29:04.276460 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:04.276433 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d72a213b-6993-4966-9527-2bb12da6bc8c-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm\" (UID: \"d72a213b-6993-4966-9527-2bb12da6bc8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm" Nov 04 03:29:04.278421 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:04.278398 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d72a213b-6993-4966-9527-2bb12da6bc8c-metrics-certs\") pod \"maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm\" (UID: \"d72a213b-6993-4966-9527-2bb12da6bc8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm" Nov 04 03:29:04.339860 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:04.339793 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg6g2\" (UniqueName: \"kubernetes.io/projected/d72a213b-6993-4966-9527-2bb12da6bc8c-kube-api-access-cg6g2\") pod \"maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm\" (UID: \"d72a213b-6993-4966-9527-2bb12da6bc8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm" Nov 04 03:29:04.781431 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:04.781386 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d72a213b-6993-4966-9527-2bb12da6bc8c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm\" (UID: \"d72a213b-6993-4966-9527-2bb12da6bc8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm" Nov 04 03:29:04.784962 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:04.784937 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d72a213b-6993-4966-9527-2bb12da6bc8c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm\" (UID: \"d72a213b-6993-4966-9527-2bb12da6bc8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm" Nov 04 03:29:04.977812 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:04.977780 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm" Nov 04 03:29:05.351491 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:05.351418 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm"] Nov 04 03:29:05.353998 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:29:05.353971 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd72a213b_6993_4966_9527_2bb12da6bc8c.slice/crio-f4b9e330c1a2dd6f32b76cbe961019dd794f18b95c922161e59249cb181994a3 WatchSource:0}: Error finding container f4b9e330c1a2dd6f32b76cbe961019dd794f18b95c922161e59249cb181994a3: Status 404 returned error can't find the container with id f4b9e330c1a2dd6f32b76cbe961019dd794f18b95c922161e59249cb181994a3 Nov 04 03:29:06.075203 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:06.075168 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm" event={"ID":"d72a213b-6993-4966-9527-2bb12da6bc8c","Type":"ContainerStarted","Data":"f4b9e330c1a2dd6f32b76cbe961019dd794f18b95c922161e59249cb181994a3"} Nov 04 03:29:07.082203 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:07.082163 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm" event={"ID":"d72a213b-6993-4966-9527-2bb12da6bc8c","Type":"ContainerStarted","Data":"98a21828b9879e2cb87296e1af60dfcb6b8649acc43c59d6712132aa4fdac8df"} Nov 04 03:29:07.082701 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:07.082234 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm" Nov 04 03:29:07.273511 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:07.273460 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm" podStartSLOduration=2.349681482 podStartE2EDuration="3.273444237s" totalImagesPullingTime="923.762755ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:29:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:29:07.27180907 +0000 UTC m=+672.546439859" watchObservedRunningTime="2026-11-04 03:29:07.273444237 +0000 UTC m=+672.548074999" Nov 04 03:29:09.080151 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:09.080112 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl"] Nov 04 03:29:09.083805 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:09.083786 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl" Nov 04 03:29:09.136710 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:09.136679 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-c5bec0-serving-cert\"" Nov 04 03:29:09.189827 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:09.189793 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl"] Nov 04 03:29:09.229677 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:09.229604 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de73559f-5ec5-4db8-9329-289c3768ac60-maas-api-tls\") pod \"maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl\" (UID: \"de73559f-5ec5-4db8-9329-289c3768ac60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl" Nov 04 03:29:09.229871 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:09.229760 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cqgx\" (UniqueName: \"kubernetes.io/projected/de73559f-5ec5-4db8-9329-289c3768ac60-kube-api-access-8cqgx\") pod \"maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl\" (UID: \"de73559f-5ec5-4db8-9329-289c3768ac60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl" Nov 04 03:29:09.229871 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:09.229820 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de73559f-5ec5-4db8-9329-289c3768ac60-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl\" (UID: \"de73559f-5ec5-4db8-9329-289c3768ac60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl" Nov 04 03:29:09.229995 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:09.229968 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de73559f-5ec5-4db8-9329-289c3768ac60-metrics-certs\") pod \"maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl\" (UID: \"de73559f-5ec5-4db8-9329-289c3768ac60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl" Nov 04 03:29:09.331399 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:09.331295 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de73559f-5ec5-4db8-9329-289c3768ac60-maas-api-tls\") pod \"maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl\" (UID: \"de73559f-5ec5-4db8-9329-289c3768ac60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl" Nov 04 03:29:09.331399 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:09.331342 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cqgx\" (UniqueName: \"kubernetes.io/projected/de73559f-5ec5-4db8-9329-289c3768ac60-kube-api-access-8cqgx\") pod \"maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl\" (UID: \"de73559f-5ec5-4db8-9329-289c3768ac60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl" Nov 04 03:29:09.331399 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:09.331374 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de73559f-5ec5-4db8-9329-289c3768ac60-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl\" (UID: \"de73559f-5ec5-4db8-9329-289c3768ac60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl" Nov 04 03:29:09.331799 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:09.331425 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de73559f-5ec5-4db8-9329-289c3768ac60-metrics-certs\") pod \"maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl\" (UID: \"de73559f-5ec5-4db8-9329-289c3768ac60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl" Nov 04 03:29:09.331799 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:29:09.331475 5953 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-c5bec0-serving-cert: secret "maas-api-e2e-worker-w3-c5bec0-serving-cert" not found Nov 04 03:29:09.331799 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:29:09.331550 5953 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de73559f-5ec5-4db8-9329-289c3768ac60-maas-api-tls podName:de73559f-5ec5-4db8-9329-289c3768ac60 nodeName:}" failed. No retries permitted until 2026-11-04 03:29:09.831527398 +0000 UTC m=+675.106158151 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/de73559f-5ec5-4db8-9329-289c3768ac60-maas-api-tls") pod "maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl" (UID: "de73559f-5ec5-4db8-9329-289c3768ac60") : secret "maas-api-e2e-worker-w3-c5bec0-serving-cert" not found Nov 04 03:29:09.332323 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:09.332300 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de73559f-5ec5-4db8-9329-289c3768ac60-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl\" (UID: \"de73559f-5ec5-4db8-9329-289c3768ac60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl" Nov 04 03:29:09.336560 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:09.336533 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de73559f-5ec5-4db8-9329-289c3768ac60-metrics-certs\") pod \"maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl\" (UID: \"de73559f-5ec5-4db8-9329-289c3768ac60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl" Nov 04 03:29:09.397306 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:09.397274 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cqgx\" (UniqueName: \"kubernetes.io/projected/de73559f-5ec5-4db8-9329-289c3768ac60-kube-api-access-8cqgx\") pod \"maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl\" (UID: \"de73559f-5ec5-4db8-9329-289c3768ac60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl" Nov 04 03:29:09.838286 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:09.838252 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de73559f-5ec5-4db8-9329-289c3768ac60-maas-api-tls\") pod \"maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl\" (UID: \"de73559f-5ec5-4db8-9329-289c3768ac60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl" Nov 04 03:29:09.840857 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:09.840832 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de73559f-5ec5-4db8-9329-289c3768ac60-maas-api-tls\") pod \"maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl\" (UID: \"de73559f-5ec5-4db8-9329-289c3768ac60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl" Nov 04 03:29:10.000147 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:10.000110 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl" Nov 04 03:29:10.376310 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:10.376269 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl"] Nov 04 03:29:10.378612 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:29:10.378581 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde73559f_5ec5_4db8_9329_289c3768ac60.slice/crio-0db16b1521369c7b3f9ca348368c2e6bd357e55ab2c02f03e63e7a3f2261bc94 WatchSource:0}: Error finding container 0db16b1521369c7b3f9ca348368c2e6bd357e55ab2c02f03e63e7a3f2261bc94: Status 404 returned error can't find the container with id 0db16b1521369c7b3f9ca348368c2e6bd357e55ab2c02f03e63e7a3f2261bc94 Nov 04 03:29:11.106025 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:11.105987 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl" event={"ID":"de73559f-5ec5-4db8-9329-289c3768ac60","Type":"ContainerStarted","Data":"0db16b1521369c7b3f9ca348368c2e6bd357e55ab2c02f03e63e7a3f2261bc94"} Nov 04 03:29:12.113273 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:12.113240 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl" event={"ID":"de73559f-5ec5-4db8-9329-289c3768ac60","Type":"ContainerStarted","Data":"fbdc963ae053144c5634b8155e11e7e7e3cbd6316b11756f8de72bf2aeb7caf7"} Nov 04 03:29:12.113741 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:12.113364 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl" Nov 04 03:29:12.243971 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:12.243917 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl" podStartSLOduration=2.397871717 podStartE2EDuration="3.24390125s" totalImagesPullingTime="846.029533ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:29:12.241758185 +0000 UTC m=+677.516388947" watchObservedRunningTime="2026-11-04 03:29:12.24390125 +0000 UTC m=+677.518532011" Nov 04 03:29:13.093514 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:13.093486 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm" Nov 04 03:29:14.066543 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:14.066519 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-10f834-6d5d8b948c-fclqf" Nov 04 03:29:16.386131 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:29:16.386090 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:29:18.124507 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:18.124469 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl" Nov 04 03:29:25.654101 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:25.654060 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-90051a-cc8656f5-6n6qz"] Nov 04 03:29:25.660692 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:25.660661 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-90051a-cc8656f5-6n6qz" Nov 04 03:29:25.734657 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:25.733691 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-90051a-cc8656f5-6n6qz"] Nov 04 03:29:25.741026 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:25.740987 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lck6v\" (UniqueName: \"kubernetes.io/projected/8772665e-c7a2-4823-be72-2b93863039ed-kube-api-access-lck6v\") pod \"payload-pre-processing-e2e-worker-w1-90051a-cc8656f5-6n6qz\" (UID: \"8772665e-c7a2-4823-be72-2b93863039ed\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-90051a-cc8656f5-6n6qz" Nov 04 03:29:25.741216 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:25.741054 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8772665e-c7a2-4823-be72-2b93863039ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-90051a-cc8656f5-6n6qz\" (UID: \"8772665e-c7a2-4823-be72-2b93863039ed\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-90051a-cc8656f5-6n6qz" Nov 04 03:29:25.763545 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:25.763516 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-90051a\"" Nov 04 03:29:25.763753 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:25.763648 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-90051a-dockercfg-snxrv\"" Nov 04 03:29:25.842345 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:25.842301 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lck6v\" (UniqueName: \"kubernetes.io/projected/8772665e-c7a2-4823-be72-2b93863039ed-kube-api-access-lck6v\") pod \"payload-pre-processing-e2e-worker-w1-90051a-cc8656f5-6n6qz\" (UID: \"8772665e-c7a2-4823-be72-2b93863039ed\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-90051a-cc8656f5-6n6qz" Nov 04 03:29:25.842524 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:25.842359 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8772665e-c7a2-4823-be72-2b93863039ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-90051a-cc8656f5-6n6qz\" (UID: \"8772665e-c7a2-4823-be72-2b93863039ed\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-90051a-cc8656f5-6n6qz" Nov 04 03:29:25.843092 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:25.843065 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8772665e-c7a2-4823-be72-2b93863039ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-90051a-cc8656f5-6n6qz\" (UID: \"8772665e-c7a2-4823-be72-2b93863039ed\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-90051a-cc8656f5-6n6qz" Nov 04 03:29:25.930739 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:25.930650 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lck6v\" (UniqueName: \"kubernetes.io/projected/8772665e-c7a2-4823-be72-2b93863039ed-kube-api-access-lck6v\") pod \"payload-pre-processing-e2e-worker-w1-90051a-cc8656f5-6n6qz\" (UID: \"8772665e-c7a2-4823-be72-2b93863039ed\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-90051a-cc8656f5-6n6qz" Nov 04 03:29:25.975052 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:25.975019 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-90051a-cc8656f5-6n6qz" Nov 04 03:29:26.371649 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:26.371605 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-90051a-cc8656f5-6n6qz"] Nov 04 03:29:26.374353 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:29:26.374324 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8772665e_c7a2_4823_be72_2b93863039ed.slice/crio-b4f99e87c8c9e5d76811e9b184efa172c38b9c2914837ba5fcc1ce3072206295 WatchSource:0}: Error finding container b4f99e87c8c9e5d76811e9b184efa172c38b9c2914837ba5fcc1ce3072206295: Status 404 returned error can't find the container with id b4f99e87c8c9e5d76811e9b184efa172c38b9c2914837ba5fcc1ce3072206295 Nov 04 03:29:27.233655 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:27.233119 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-90051a-cc8656f5-6n6qz" event={"ID":"8772665e-c7a2-4823-be72-2b93863039ed","Type":"ContainerStarted","Data":"70f4a7e8d22543b5baa574924b0f9d451d9fd35ca825fb8b4d426214e609cab7"} Nov 04 03:29:27.233655 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:27.233187 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-90051a-cc8656f5-6n6qz" event={"ID":"8772665e-c7a2-4823-be72-2b93863039ed","Type":"ContainerStarted","Data":"b4f99e87c8c9e5d76811e9b184efa172c38b9c2914837ba5fcc1ce3072206295"} Nov 04 03:29:27.234851 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:27.234759 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-90051a-cc8656f5-6n6qz" Nov 04 03:29:27.352459 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:27.352404 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-90051a-cc8656f5-6n6qz" podStartSLOduration=2.352390022 podStartE2EDuration="2.352390022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:29:27.349885271 +0000 UTC m=+692.624516033" watchObservedRunningTime="2026-11-04 03:29:27.352390022 +0000 UTC m=+692.627020784" Nov 04 03:29:30.185952 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:30.185542 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz"] Nov 04 03:29:30.190453 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:30.190431 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz" Nov 04 03:29:30.288269 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:30.288232 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f4748f7-aaab-4945-9bb1-6e7ae7c287cc-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz\" (UID: \"3f4748f7-aaab-4945-9bb1-6e7ae7c287cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz" Nov 04 03:29:30.288269 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:30.288273 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f4748f7-aaab-4945-9bb1-6e7ae7c287cc-metrics-certs\") pod \"maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz\" (UID: \"3f4748f7-aaab-4945-9bb1-6e7ae7c287cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz" Nov 04 03:29:30.288550 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:30.288464 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grrt6\" (UniqueName: \"kubernetes.io/projected/3f4748f7-aaab-4945-9bb1-6e7ae7c287cc-kube-api-access-grrt6\") pod \"maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz\" (UID: \"3f4748f7-aaab-4945-9bb1-6e7ae7c287cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz" Nov 04 03:29:30.288550 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:30.288501 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f4748f7-aaab-4945-9bb1-6e7ae7c287cc-maas-api-tls\") pod \"maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz\" (UID: \"3f4748f7-aaab-4945-9bb1-6e7ae7c287cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz" Nov 04 03:29:30.295952 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:30.295918 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-36ccd3-serving-cert\"" Nov 04 03:29:30.298963 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:30.298933 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz"] Nov 04 03:29:30.389605 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:30.389570 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grrt6\" (UniqueName: \"kubernetes.io/projected/3f4748f7-aaab-4945-9bb1-6e7ae7c287cc-kube-api-access-grrt6\") pod \"maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz\" (UID: \"3f4748f7-aaab-4945-9bb1-6e7ae7c287cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz" Nov 04 03:29:30.389799 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:30.389630 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f4748f7-aaab-4945-9bb1-6e7ae7c287cc-maas-api-tls\") pod \"maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz\" (UID: \"3f4748f7-aaab-4945-9bb1-6e7ae7c287cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz" Nov 04 03:29:30.389799 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:30.389698 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f4748f7-aaab-4945-9bb1-6e7ae7c287cc-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz\" (UID: \"3f4748f7-aaab-4945-9bb1-6e7ae7c287cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz" Nov 04 03:29:30.389799 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:30.389722 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f4748f7-aaab-4945-9bb1-6e7ae7c287cc-metrics-certs\") pod \"maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz\" (UID: \"3f4748f7-aaab-4945-9bb1-6e7ae7c287cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz" Nov 04 03:29:30.389799 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:29:30.389772 5953 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-36ccd3-serving-cert: secret "maas-api-e2e-worker-w0-36ccd3-serving-cert" not found Nov 04 03:29:30.389963 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:29:30.389843 5953 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f4748f7-aaab-4945-9bb1-6e7ae7c287cc-maas-api-tls podName:3f4748f7-aaab-4945-9bb1-6e7ae7c287cc nodeName:}" failed. No retries permitted until 2026-11-04 03:29:30.889823198 +0000 UTC m=+696.164453942 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3f4748f7-aaab-4945-9bb1-6e7ae7c287cc-maas-api-tls") pod "maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz" (UID: "3f4748f7-aaab-4945-9bb1-6e7ae7c287cc") : secret "maas-api-e2e-worker-w0-36ccd3-serving-cert" not found Nov 04 03:29:30.390430 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:30.390412 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f4748f7-aaab-4945-9bb1-6e7ae7c287cc-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz\" (UID: \"3f4748f7-aaab-4945-9bb1-6e7ae7c287cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz" Nov 04 03:29:30.392482 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:30.392459 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f4748f7-aaab-4945-9bb1-6e7ae7c287cc-metrics-certs\") pod \"maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz\" (UID: \"3f4748f7-aaab-4945-9bb1-6e7ae7c287cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz" Nov 04 03:29:30.452907 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:30.452838 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grrt6\" (UniqueName: \"kubernetes.io/projected/3f4748f7-aaab-4945-9bb1-6e7ae7c287cc-kube-api-access-grrt6\") pod \"maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz\" (UID: \"3f4748f7-aaab-4945-9bb1-6e7ae7c287cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz" Nov 04 03:29:30.896351 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:30.896317 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f4748f7-aaab-4945-9bb1-6e7ae7c287cc-maas-api-tls\") pod \"maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz\" (UID: \"3f4748f7-aaab-4945-9bb1-6e7ae7c287cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz" Nov 04 03:29:30.899080 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:30.899054 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f4748f7-aaab-4945-9bb1-6e7ae7c287cc-maas-api-tls\") pod \"maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz\" (UID: \"3f4748f7-aaab-4945-9bb1-6e7ae7c287cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz" Nov 04 03:29:31.106058 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:31.106016 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz" Nov 04 03:29:31.482758 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:31.482733 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz"] Nov 04 03:29:31.485016 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:29:31.484971 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f4748f7_aaab_4945_9bb1_6e7ae7c287cc.slice/crio-c21de1c15f54e24c84667d72302f1417db8db6c3bdbaf58ee1b4c6e1b752509c WatchSource:0}: Error finding container c21de1c15f54e24c84667d72302f1417db8db6c3bdbaf58ee1b4c6e1b752509c: Status 404 returned error can't find the container with id c21de1c15f54e24c84667d72302f1417db8db6c3bdbaf58ee1b4c6e1b752509c Nov 04 03:29:32.264979 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:32.264944 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz" event={"ID":"3f4748f7-aaab-4945-9bb1-6e7ae7c287cc","Type":"ContainerStarted","Data":"c21de1c15f54e24c84667d72302f1417db8db6c3bdbaf58ee1b4c6e1b752509c"} Nov 04 03:29:32.266595 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:32.266571 5953 generic.go:356] "Generic (PLEG): container finished" podID="cdfcbe1f-229d-4c24-9e85-01e59dac657a" containerID="15061a35835d7a806d91781392f465eb88d2d8dad663d6687aa49a7db223b4b7" exitCode=0 Nov 04 03:29:32.266763 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:32.266651 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" event={"ID":"cdfcbe1f-229d-4c24-9e85-01e59dac657a","Type":"ContainerDied","Data":"15061a35835d7a806d91781392f465eb88d2d8dad663d6687aa49a7db223b4b7"} Nov 04 03:29:32.267071 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:32.267047 5953 scope.go:122] "RemoveContainer" containerID="15061a35835d7a806d91781392f465eb88d2d8dad663d6687aa49a7db223b4b7" Nov 04 03:29:32.268400 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:32.268373 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:29:32.268493 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:32.268457 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:29:32.268553 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:32.268505 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:29:33.275057 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:33.275012 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz" event={"ID":"3f4748f7-aaab-4945-9bb1-6e7ae7c287cc","Type":"ContainerStarted","Data":"4d4e527beee7b781319aabb25930447914163b9ddca7a1bcca13ad32a0c43037"} Nov 04 03:29:33.275557 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:33.275124 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz" Nov 04 03:29:33.277153 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:33.277128 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" event={"ID":"cdfcbe1f-229d-4c24-9e85-01e59dac657a","Type":"ContainerStarted","Data":"fff3383cc2cb66f3d65434a859143631e9106ed7b51ce4bf62741ef0826c0a52"} Nov 04 03:29:33.395369 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:33.395301 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz" podStartSLOduration=2.573351665 podStartE2EDuration="3.395282077s" totalImagesPullingTime="821.930412ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:29:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:29:33.392732296 +0000 UTC m=+698.667363061" watchObservedRunningTime="2026-11-04 03:29:33.395282077 +0000 UTC m=+698.669912840" Nov 04 03:29:33.825644 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:33.825041 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" Nov 04 03:29:33.825644 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:33.825179 5953 patch_prober.go:29] interesting pod/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 04 03:29:33.825644 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:33.825236 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" podUID="cdfcbe1f-229d-4c24-9e85-01e59dac657a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 04 03:29:34.819059 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:34.819020 5953 patch_prober.go:29] interesting pod/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 04 03:29:34.819705 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:34.819671 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" podUID="cdfcbe1f-229d-4c24-9e85-01e59dac657a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 04 03:29:35.823644 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:35.819064 5953 patch_prober.go:29] interesting pod/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 04 03:29:35.823644 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:35.819171 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" podUID="cdfcbe1f-229d-4c24-9e85-01e59dac657a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 04 03:29:36.827648 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:36.826138 5953 patch_prober.go:29] interesting pod/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 04 03:29:36.827648 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:36.826200 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" podUID="cdfcbe1f-229d-4c24-9e85-01e59dac657a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 04 03:29:37.827834 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:37.827783 5953 patch_prober.go:29] interesting pod/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 04 03:29:37.828288 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:37.827860 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" podUID="cdfcbe1f-229d-4c24-9e85-01e59dac657a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 04 03:29:38.818246 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:38.818211 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" Nov 04 03:29:38.818555 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:38.818533 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" Nov 04 03:29:38.819332 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:38.819315 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" Nov 04 03:29:39.249723 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:39.249635 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-90051a-cc8656f5-6n6qz" Nov 04 03:29:39.289233 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:29:39.289199 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz" Nov 04 03:29:46.387282 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:29:46.387244 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:30:00.173434 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:00.173398 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896050-r27b6"] Nov 04 03:30:00.178517 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:00.178492 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896050-r27b6" Nov 04 03:30:00.229602 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:00.229561 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98127a25-55e0-4e47-a638-aad74a326fe4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896050-r27b6\" (UID: \"98127a25-55e0-4e47-a638-aad74a326fe4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896050-r27b6" Nov 04 03:30:00.229819 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:00.229642 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcdgv\" (UniqueName: \"kubernetes.io/projected/98127a25-55e0-4e47-a638-aad74a326fe4-kube-api-access-kcdgv\") pod \"maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896050-r27b6\" (UID: \"98127a25-55e0-4e47-a638-aad74a326fe4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896050-r27b6" Nov 04 03:30:00.231461 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:00.231437 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hlrnm\"" Nov 04 03:30:00.285190 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:00.285149 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896050-r27b6"] Nov 04 03:30:00.331133 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:00.331094 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98127a25-55e0-4e47-a638-aad74a326fe4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896050-r27b6\" (UID: \"98127a25-55e0-4e47-a638-aad74a326fe4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896050-r27b6" Nov 04 03:30:00.331324 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:00.331152 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcdgv\" (UniqueName: \"kubernetes.io/projected/98127a25-55e0-4e47-a638-aad74a326fe4-kube-api-access-kcdgv\") pod \"maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896050-r27b6\" (UID: \"98127a25-55e0-4e47-a638-aad74a326fe4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896050-r27b6" Nov 04 03:30:00.332041 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:00.332014 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98127a25-55e0-4e47-a638-aad74a326fe4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896050-r27b6\" (UID: \"98127a25-55e0-4e47-a638-aad74a326fe4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896050-r27b6" Nov 04 03:30:00.394207 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:00.394178 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcdgv\" (UniqueName: \"kubernetes.io/projected/98127a25-55e0-4e47-a638-aad74a326fe4-kube-api-access-kcdgv\") pod \"maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896050-r27b6\" (UID: \"98127a25-55e0-4e47-a638-aad74a326fe4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896050-r27b6" Nov 04 03:30:00.497457 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:00.497360 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896050-r27b6" Nov 04 03:30:00.879223 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:00.879186 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896050-r27b6"] Nov 04 03:30:00.881848 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:30:00.881812 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98127a25_55e0_4e47_a638_aad74a326fe4.slice/crio-bba5d832d478b8dced06e4cc81598026727e929fff08c08960083676f6bea366 WatchSource:0}: Error finding container bba5d832d478b8dced06e4cc81598026727e929fff08c08960083676f6bea366: Status 404 returned error can't find the container with id bba5d832d478b8dced06e4cc81598026727e929fff08c08960083676f6bea366 Nov 04 03:30:01.478303 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:01.478272 5953 generic.go:356] "Generic (PLEG): container finished" podID="e9d98901-eb52-4bd0-a031-640623bd4272" containerID="9494120f489a4d24f206df071b044ca6655db40dd6f4ac65f423685e8added78" exitCode=0 Nov 04 03:30:01.478803 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:01.478343 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" event={"ID":"e9d98901-eb52-4bd0-a031-640623bd4272","Type":"ContainerDied","Data":"9494120f489a4d24f206df071b044ca6655db40dd6f4ac65f423685e8added78"} Nov 04 03:30:01.478803 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:01.478787 5953 scope.go:122] "RemoveContainer" containerID="9494120f489a4d24f206df071b044ca6655db40dd6f4ac65f423685e8added78" Nov 04 03:30:01.479909 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:01.479879 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896050-r27b6" event={"ID":"98127a25-55e0-4e47-a638-aad74a326fe4","Type":"ContainerStarted","Data":"bba5d832d478b8dced06e4cc81598026727e929fff08c08960083676f6bea366"} Nov 04 03:30:01.480241 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:01.480216 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:30:01.480301 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:01.480288 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:30:01.480336 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:01.480329 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:30:02.488273 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:02.488223 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" event={"ID":"e9d98901-eb52-4bd0-a031-640623bd4272","Type":"ContainerStarted","Data":"631110178fbc6ef1a26586f6ca6109f7d52e464849a0ed7729fe4bf127a1a46b"} Nov 04 03:30:02.793564 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:02.793527 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" Nov 04 03:30:02.794820 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:02.794795 5953 patch_prober.go:29] interesting pod/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 04 03:30:02.794926 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:02.794853 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" podUID="e9d98901-eb52-4bd0-a031-640623bd4272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 04 03:30:03.793458 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:03.793364 5953 patch_prober.go:29] interesting pod/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 04 03:30:03.793458 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:03.793443 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" podUID="e9d98901-eb52-4bd0-a031-640623bd4272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 04 03:30:04.063659 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:04.063561 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" Nov 04 03:30:04.793606 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:04.793572 5953 patch_prober.go:29] interesting pod/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 04 03:30:04.794115 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:04.793668 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" podUID="e9d98901-eb52-4bd0-a031-640623bd4272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 04 03:30:05.804060 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:05.804026 5953 patch_prober.go:29] interesting pod/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 04 03:30:05.804683 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:05.804649 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" podUID="e9d98901-eb52-4bd0-a031-640623bd4272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 04 03:30:06.539978 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:06.539937 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896050-r27b6" event={"ID":"98127a25-55e0-4e47-a638-aad74a326fe4","Type":"ContainerStarted","Data":"a296a0ab8de27cd001f06d861557de2fbebf537f66078dc5ae704675a506027e"} Nov 04 03:30:06.655886 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:06.655827 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896050-r27b6" podStartSLOduration=1.977064591 podStartE2EDuration="6.655813855s" totalImagesPullingTime="4.678749264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:30:06.655220065 +0000 UTC m=+731.929850845" watchObservedRunningTime="2026-11-04 03:30:06.655813855 +0000 UTC m=+731.930444617" Nov 04 03:30:06.793490 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:06.793399 5953 patch_prober.go:29] interesting pod/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 04 03:30:06.793490 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:06.793469 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" podUID="e9d98901-eb52-4bd0-a031-640623bd4272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 04 03:30:07.547391 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:07.547356 5953 generic.go:356] "Generic (PLEG): container finished" podID="98127a25-55e0-4e47-a638-aad74a326fe4" containerID="a296a0ab8de27cd001f06d861557de2fbebf537f66078dc5ae704675a506027e" exitCode=60 Nov 04 03:30:07.547871 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:07.547448 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896050-r27b6" event={"ID":"98127a25-55e0-4e47-a638-aad74a326fe4","Type":"ContainerDied","Data":"a296a0ab8de27cd001f06d861557de2fbebf537f66078dc5ae704675a506027e"} Nov 04 03:30:07.547871 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:07.547737 5953 scope.go:122] "RemoveContainer" containerID="a296a0ab8de27cd001f06d861557de2fbebf537f66078dc5ae704675a506027e" Nov 04 03:30:07.798264 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:07.798184 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" Nov 04 03:30:07.799382 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:07.799357 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" Nov 04 03:30:08.556243 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:08.556211 5953 generic.go:356] "Generic (PLEG): container finished" podID="98127a25-55e0-4e47-a638-aad74a326fe4" containerID="0782cb8a64386bf44c940aa38985eb77d143d3de2157c61dfbdf9ffad5c835e8" exitCode=60 Nov 04 03:30:08.556764 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:08.556301 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896050-r27b6" event={"ID":"98127a25-55e0-4e47-a638-aad74a326fe4","Type":"ContainerDied","Data":"0782cb8a64386bf44c940aa38985eb77d143d3de2157c61dfbdf9ffad5c835e8"} Nov 04 03:30:08.556764 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:08.556369 5953 scope.go:122] "RemoveContainer" containerID="a296a0ab8de27cd001f06d861557de2fbebf537f66078dc5ae704675a506027e" Nov 04 03:30:08.556764 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:08.556595 5953 scope.go:122] "RemoveContainer" containerID="0782cb8a64386bf44c940aa38985eb77d143d3de2157c61dfbdf9ffad5c835e8" Nov 04 03:30:08.556968 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:30:08.556945 5953 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896050-r27b6_odh-ai-gateway-infra(98127a25-55e0-4e47-a638-aad74a326fe4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896050-r27b6" podUID="98127a25-55e0-4e47-a638-aad74a326fe4" Nov 04 03:30:09.564028 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:09.563994 5953 scope.go:122] "RemoveContainer" containerID="0782cb8a64386bf44c940aa38985eb77d143d3de2157c61dfbdf9ffad5c835e8" Nov 04 03:30:09.564488 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:30:09.564331 5953 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896050-r27b6_odh-ai-gateway-infra(98127a25-55e0-4e47-a638-aad74a326fe4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896050-r27b6" podUID="98127a25-55e0-4e47-a638-aad74a326fe4" Nov 04 03:30:16.384552 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:30:16.384512 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:30:22.849850 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:22.849818 5953 scope.go:122] "RemoveContainer" containerID="0782cb8a64386bf44c940aa38985eb77d143d3de2157c61dfbdf9ffad5c835e8" Nov 04 03:30:23.647908 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:23.647869 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896050-r27b6" event={"ID":"98127a25-55e0-4e47-a638-aad74a326fe4","Type":"ContainerStarted","Data":"2f494078078b48b63c87781067298d9248a9d5b1ac96f92989d8ce6d94dc30a8"} Nov 04 03:30:24.655565 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:24.655528 5953 generic.go:356] "Generic (PLEG): container finished" podID="98127a25-55e0-4e47-a638-aad74a326fe4" containerID="2f494078078b48b63c87781067298d9248a9d5b1ac96f92989d8ce6d94dc30a8" exitCode=60 Nov 04 03:30:24.655990 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:24.655676 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896050-r27b6" event={"ID":"98127a25-55e0-4e47-a638-aad74a326fe4","Type":"ContainerDied","Data":"2f494078078b48b63c87781067298d9248a9d5b1ac96f92989d8ce6d94dc30a8"} Nov 04 03:30:24.655990 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:24.655770 5953 scope.go:122] "RemoveContainer" containerID="0782cb8a64386bf44c940aa38985eb77d143d3de2157c61dfbdf9ffad5c835e8" Nov 04 03:30:24.656199 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:24.656184 5953 scope.go:122] "RemoveContainer" containerID="2f494078078b48b63c87781067298d9248a9d5b1ac96f92989d8ce6d94dc30a8" Nov 04 03:30:24.656464 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:30:24.656444 5953 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896050-r27b6_odh-ai-gateway-infra(98127a25-55e0-4e47-a638-aad74a326fe4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896050-r27b6" podUID="98127a25-55e0-4e47-a638-aad74a326fe4" Nov 04 03:30:24.783225 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:24.783190 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896050-r27b6"] Nov 04 03:30:25.805957 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:25.805931 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896050-r27b6" Nov 04 03:30:25.867544 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:25.867516 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98127a25-55e0-4e47-a638-aad74a326fe4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98127a25-55e0-4e47-a638-aad74a326fe4-maas-api-service-ca\") pod \"98127a25-55e0-4e47-a638-aad74a326fe4\" (UID: \"98127a25-55e0-4e47-a638-aad74a326fe4\") " Nov 04 03:30:25.867744 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:25.867557 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98127a25-55e0-4e47-a638-aad74a326fe4-kube-api-access-kcdgv\" (UniqueName: \"kubernetes.io/projected/98127a25-55e0-4e47-a638-aad74a326fe4-kube-api-access-kcdgv\") pod \"98127a25-55e0-4e47-a638-aad74a326fe4\" (UID: \"98127a25-55e0-4e47-a638-aad74a326fe4\") " Nov 04 03:30:25.867907 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:25.867883 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98127a25-55e0-4e47-a638-aad74a326fe4-maas-api-service-ca" pod "98127a25-55e0-4e47-a638-aad74a326fe4" (UID: "98127a25-55e0-4e47-a638-aad74a326fe4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:30:25.869859 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:25.869836 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98127a25-55e0-4e47-a638-aad74a326fe4-kube-api-access-kcdgv" pod "98127a25-55e0-4e47-a638-aad74a326fe4" (UID: "98127a25-55e0-4e47-a638-aad74a326fe4"). InnerVolumeSpecName "kube-api-access-kcdgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:30:25.969744 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:25.969586 5953 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98127a25-55e0-4e47-a638-aad74a326fe4-maas-api-service-ca\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:30:25.969744 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:25.969664 5953 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcdgv\" (UniqueName: \"kubernetes.io/projected/98127a25-55e0-4e47-a638-aad74a326fe4-kube-api-access-kcdgv\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:30:26.669634 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:26.669600 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896050-r27b6" Nov 04 03:30:26.669850 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:26.669662 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896050-r27b6" event={"ID":"98127a25-55e0-4e47-a638-aad74a326fe4","Type":"ContainerDied","Data":"bba5d832d478b8dced06e4cc81598026727e929fff08c08960083676f6bea366"} Nov 04 03:30:26.669850 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:26.669738 5953 scope.go:122] "RemoveContainer" containerID="2f494078078b48b63c87781067298d9248a9d5b1ac96f92989d8ce6d94dc30a8" Nov 04 03:30:26.835018 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:26.834985 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896050-r27b6"] Nov 04 03:30:26.884889 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:26.884860 5953 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896050-r27b6"] Nov 04 03:30:28.856317 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:28.856285 5953 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98127a25-55e0-4e47-a638-aad74a326fe4" path="/var/lib/kubelet/pods/98127a25-55e0-4e47-a638-aad74a326fe4/volumes" Nov 04 03:30:39.653489 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:39.653453 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w"] Nov 04 03:30:39.654200 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:39.654178 5953 cpu_manager.go:414] "containerMap: removing container" podUID="98127a25-55e0-4e47-a638-aad74a326fe4" containerName="cleanup" Nov 04 03:30:39.654301 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:39.654201 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98127a25-55e0-4e47-a638-aad74a326fe4" containerName="cleanup" Nov 04 03:30:39.654301 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:39.654216 5953 cpu_manager.go:414] "containerMap: removing container" podUID="98127a25-55e0-4e47-a638-aad74a326fe4" containerName="cleanup" Nov 04 03:30:39.654301 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:39.654223 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98127a25-55e0-4e47-a638-aad74a326fe4" containerName="cleanup" Nov 04 03:30:39.654301 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:39.654250 5953 cpu_manager.go:414] "containerMap: removing container" podUID="98127a25-55e0-4e47-a638-aad74a326fe4" containerName="cleanup" Nov 04 03:30:39.654301 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:39.654258 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98127a25-55e0-4e47-a638-aad74a326fe4" containerName="cleanup" Nov 04 03:30:39.654542 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:39.654421 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="98127a25-55e0-4e47-a638-aad74a326fe4" containerName="cleanup" Nov 04 03:30:39.654542 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:39.654444 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="98127a25-55e0-4e47-a638-aad74a326fe4" containerName="cleanup" Nov 04 03:30:39.658171 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:39.658147 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" Nov 04 03:30:39.711261 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:39.711236 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-10f834-openshift-default-dockercfg-4d9rk\"" Nov 04 03:30:39.765205 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:39.765128 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w"] Nov 04 03:30:39.823852 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:39.823811 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e573d41f-40a7-4e92-880d-a0204b14633a-istio-podinfo\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w\" (UID: \"e573d41f-40a7-4e92-880d-a0204b14633a\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" Nov 04 03:30:39.823852 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:39.823850 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e573d41f-40a7-4e92-880d-a0204b14633a-credential-socket\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w\" (UID: \"e573d41f-40a7-4e92-880d-a0204b14633a\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" Nov 04 03:30:39.824068 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:39.823869 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e573d41f-40a7-4e92-880d-a0204b14633a-istio-envoy\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w\" (UID: \"e573d41f-40a7-4e92-880d-a0204b14633a\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" Nov 04 03:30:39.824068 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:39.824014 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e573d41f-40a7-4e92-880d-a0204b14633a-istio-data\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w\" (UID: \"e573d41f-40a7-4e92-880d-a0204b14633a\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" Nov 04 03:30:39.824137 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:39.824075 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e573d41f-40a7-4e92-880d-a0204b14633a-workload-certs\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w\" (UID: \"e573d41f-40a7-4e92-880d-a0204b14633a\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" Nov 04 03:30:39.824137 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:39.824120 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e573d41f-40a7-4e92-880d-a0204b14633a-workload-socket\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w\" (UID: \"e573d41f-40a7-4e92-880d-a0204b14633a\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" Nov 04 03:30:39.824198 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:39.824145 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e573d41f-40a7-4e92-880d-a0204b14633a-istio-token\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w\" (UID: \"e573d41f-40a7-4e92-880d-a0204b14633a\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" Nov 04 03:30:39.824198 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:39.824182 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e573d41f-40a7-4e92-880d-a0204b14633a-istiod-ca-cert\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w\" (UID: \"e573d41f-40a7-4e92-880d-a0204b14633a\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" Nov 04 03:30:39.824265 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:39.824243 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27s7q\" (UniqueName: \"kubernetes.io/projected/e573d41f-40a7-4e92-880d-a0204b14633a-kube-api-access-27s7q\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w\" (UID: \"e573d41f-40a7-4e92-880d-a0204b14633a\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" Nov 04 03:30:39.925454 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:39.925369 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e573d41f-40a7-4e92-880d-a0204b14633a-workload-certs\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w\" (UID: \"e573d41f-40a7-4e92-880d-a0204b14633a\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" Nov 04 03:30:39.925454 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:39.925408 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e573d41f-40a7-4e92-880d-a0204b14633a-workload-socket\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w\" (UID: \"e573d41f-40a7-4e92-880d-a0204b14633a\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" Nov 04 03:30:39.925454 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:39.925431 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e573d41f-40a7-4e92-880d-a0204b14633a-istio-token\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w\" (UID: \"e573d41f-40a7-4e92-880d-a0204b14633a\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" Nov 04 03:30:39.925454 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:39.925451 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e573d41f-40a7-4e92-880d-a0204b14633a-istiod-ca-cert\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w\" (UID: \"e573d41f-40a7-4e92-880d-a0204b14633a\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" Nov 04 03:30:39.925759 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:39.925520 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27s7q\" (UniqueName: \"kubernetes.io/projected/e573d41f-40a7-4e92-880d-a0204b14633a-kube-api-access-27s7q\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w\" (UID: \"e573d41f-40a7-4e92-880d-a0204b14633a\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" Nov 04 03:30:39.925759 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:39.925564 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e573d41f-40a7-4e92-880d-a0204b14633a-istio-podinfo\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w\" (UID: \"e573d41f-40a7-4e92-880d-a0204b14633a\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" Nov 04 03:30:39.925759 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:39.925587 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e573d41f-40a7-4e92-880d-a0204b14633a-credential-socket\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w\" (UID: \"e573d41f-40a7-4e92-880d-a0204b14633a\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" Nov 04 03:30:39.925759 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:39.925612 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e573d41f-40a7-4e92-880d-a0204b14633a-istio-envoy\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w\" (UID: \"e573d41f-40a7-4e92-880d-a0204b14633a\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" Nov 04 03:30:39.925759 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:39.925707 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e573d41f-40a7-4e92-880d-a0204b14633a-istio-data\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w\" (UID: \"e573d41f-40a7-4e92-880d-a0204b14633a\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" Nov 04 03:30:39.925942 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:39.925919 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e573d41f-40a7-4e92-880d-a0204b14633a-workload-certs\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w\" (UID: \"e573d41f-40a7-4e92-880d-a0204b14633a\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" Nov 04 03:30:39.926002 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:39.925979 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e573d41f-40a7-4e92-880d-a0204b14633a-workload-socket\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w\" (UID: \"e573d41f-40a7-4e92-880d-a0204b14633a\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" Nov 04 03:30:39.926068 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:39.926053 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e573d41f-40a7-4e92-880d-a0204b14633a-istio-data\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w\" (UID: \"e573d41f-40a7-4e92-880d-a0204b14633a\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" Nov 04 03:30:39.926234 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:39.926213 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e573d41f-40a7-4e92-880d-a0204b14633a-credential-socket\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w\" (UID: \"e573d41f-40a7-4e92-880d-a0204b14633a\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" Nov 04 03:30:39.926323 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:39.926297 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e573d41f-40a7-4e92-880d-a0204b14633a-istiod-ca-cert\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w\" (UID: \"e573d41f-40a7-4e92-880d-a0204b14633a\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" Nov 04 03:30:39.928234 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:39.928211 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e573d41f-40a7-4e92-880d-a0204b14633a-istio-envoy\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w\" (UID: \"e573d41f-40a7-4e92-880d-a0204b14633a\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" Nov 04 03:30:39.928358 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:39.928337 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e573d41f-40a7-4e92-880d-a0204b14633a-istio-podinfo\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w\" (UID: \"e573d41f-40a7-4e92-880d-a0204b14633a\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" Nov 04 03:30:39.987109 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:39.987072 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e573d41f-40a7-4e92-880d-a0204b14633a-istio-token\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w\" (UID: \"e573d41f-40a7-4e92-880d-a0204b14633a\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" Nov 04 03:30:39.987254 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:39.987195 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27s7q\" (UniqueName: \"kubernetes.io/projected/e573d41f-40a7-4e92-880d-a0204b14633a-kube-api-access-27s7q\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w\" (UID: \"e573d41f-40a7-4e92-880d-a0204b14633a\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" Nov 04 03:30:40.271976 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:40.271938 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" Nov 04 03:30:40.701372 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:40.701330 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w"] Nov 04 03:30:40.703898 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:30:40.703872 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode573d41f_40a7_4e92_880d_a0204b14633a.slice/crio-0c458d67f8a3d2a25e336eeb8f3832cb691ff72007a164387aa89d6f791d474d WatchSource:0}: Error finding container 0c458d67f8a3d2a25e336eeb8f3832cb691ff72007a164387aa89d6f791d474d: Status 404 returned error can't find the container with id 0c458d67f8a3d2a25e336eeb8f3832cb691ff72007a164387aa89d6f791d474d Nov 04 03:30:40.706237 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:40.706208 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:30:40.706301 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:40.706269 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:30:40.706301 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:40.706297 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:30:40.754163 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:40.754131 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" event={"ID":"e573d41f-40a7-4e92-880d-a0204b14633a","Type":"ContainerStarted","Data":"0c458d67f8a3d2a25e336eeb8f3832cb691ff72007a164387aa89d6f791d474d"} Nov 04 03:30:41.761081 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:41.761043 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" event={"ID":"e573d41f-40a7-4e92-880d-a0204b14633a","Type":"ContainerStarted","Data":"e02083754d8c50a5bf60ad89092401202602e4d2e292bf9ad57b909627192e31"} Nov 04 03:30:41.993945 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:41.993894 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" podStartSLOduration=2.99387428 podStartE2EDuration="2.99387428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:30:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:30:41.942566504 +0000 UTC m=+767.217197268" watchObservedRunningTime="2026-11-04 03:30:41.99387428 +0000 UTC m=+767.268505043" Nov 04 03:30:42.273127 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:42.273091 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" Nov 04 03:30:42.274842 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:42.274816 5953 patch_prober.go:29] interesting pod/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 04 03:30:42.274982 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:42.274861 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" podUID="e573d41f-40a7-4e92-880d-a0204b14633a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 04 03:30:43.272232 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:43.272198 5953 patch_prober.go:29] interesting pod/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 04 03:30:43.272736 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:43.272251 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" podUID="e573d41f-40a7-4e92-880d-a0204b14633a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 04 03:30:44.272315 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:44.272278 5953 patch_prober.go:29] interesting pod/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 04 03:30:44.272758 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:44.272339 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" podUID="e573d41f-40a7-4e92-880d-a0204b14633a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 04 03:30:45.272504 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:45.272468 5953 patch_prober.go:29] interesting pod/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 04 03:30:45.272929 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:45.272537 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" podUID="e573d41f-40a7-4e92-880d-a0204b14633a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 04 03:30:46.272886 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:46.272853 5953 patch_prober.go:29] interesting pod/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 04 03:30:46.273286 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:46.272909 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" podUID="e573d41f-40a7-4e92-880d-a0204b14633a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 04 03:30:46.385938 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:30:46.385892 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:30:47.499756 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:47.499697 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" Nov 04 03:30:47.500955 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:47.500923 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" Nov 04 03:30:47.501742 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:30:47.501717 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" Nov 04 03:31:16.384585 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:31:16.384544 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:31:43.388843 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:31:43.388801 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-36ccd3/premium-w0-36ccd3-kserve-78d464b6d8-snzsz"] Nov 04 03:31:43.389759 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:31:43.389709 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="98127a25-55e0-4e47-a638-aad74a326fe4" containerName="cleanup" Nov 04 03:31:43.393302 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:31:43.393275 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-36ccd3/premium-w0-36ccd3-kserve-78d464b6d8-snzsz" Nov 04 03:31:43.496405 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:31:43.496367 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-36ccd3\"/\"openshift-service-ca.crt\"" Nov 04 03:31:43.496405 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:31:43.496373 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-36ccd3\"/\"kube-root-ca.crt\"" Nov 04 03:31:43.496612 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:31:43.496370 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-36ccd3\"/\"default-dockercfg-t28c2\"" Nov 04 03:31:43.496612 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:31:43.496404 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-36ccd3\"/\"premium-w0-36ccd3-kserve-self-signed-certs\"" Nov 04 03:31:43.498413 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:31:43.498393 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-36ccd3/premium-w0-36ccd3-kserve-78d464b6d8-snzsz"] Nov 04 03:31:43.522952 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:31:43.522923 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtqtf\" (UniqueName: \"kubernetes.io/projected/371e82db-7c6c-411d-8f38-988f76bcf821-kube-api-access-rtqtf\") pod \"premium-w0-36ccd3-kserve-78d464b6d8-snzsz\" (UID: \"371e82db-7c6c-411d-8f38-988f76bcf821\") " pod="e2e-models-e2e-worker-w0-36ccd3/premium-w0-36ccd3-kserve-78d464b6d8-snzsz" Nov 04 03:31:43.523123 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:31:43.522967 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/371e82db-7c6c-411d-8f38-988f76bcf821-dshm\") pod \"premium-w0-36ccd3-kserve-78d464b6d8-snzsz\" (UID: \"371e82db-7c6c-411d-8f38-988f76bcf821\") " pod="e2e-models-e2e-worker-w0-36ccd3/premium-w0-36ccd3-kserve-78d464b6d8-snzsz" Nov 04 03:31:43.523123 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:31:43.522988 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/371e82db-7c6c-411d-8f38-988f76bcf821-model-cache\") pod \"premium-w0-36ccd3-kserve-78d464b6d8-snzsz\" (UID: \"371e82db-7c6c-411d-8f38-988f76bcf821\") " pod="e2e-models-e2e-worker-w0-36ccd3/premium-w0-36ccd3-kserve-78d464b6d8-snzsz" Nov 04 03:31:43.523209 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:31:43.523190 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/371e82db-7c6c-411d-8f38-988f76bcf821-home\") pod \"premium-w0-36ccd3-kserve-78d464b6d8-snzsz\" (UID: \"371e82db-7c6c-411d-8f38-988f76bcf821\") " pod="e2e-models-e2e-worker-w0-36ccd3/premium-w0-36ccd3-kserve-78d464b6d8-snzsz" Nov 04 03:31:43.523248 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:31:43.523231 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/371e82db-7c6c-411d-8f38-988f76bcf821-tmp-dir\") pod \"premium-w0-36ccd3-kserve-78d464b6d8-snzsz\" (UID: \"371e82db-7c6c-411d-8f38-988f76bcf821\") " pod="e2e-models-e2e-worker-w0-36ccd3/premium-w0-36ccd3-kserve-78d464b6d8-snzsz" Nov 04 03:31:43.523306 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:31:43.523258 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/371e82db-7c6c-411d-8f38-988f76bcf821-tls-certs\") pod \"premium-w0-36ccd3-kserve-78d464b6d8-snzsz\" (UID: \"371e82db-7c6c-411d-8f38-988f76bcf821\") " pod="e2e-models-e2e-worker-w0-36ccd3/premium-w0-36ccd3-kserve-78d464b6d8-snzsz" Nov 04 03:31:43.624637 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:31:43.624521 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/371e82db-7c6c-411d-8f38-988f76bcf821-dshm\") pod \"premium-w0-36ccd3-kserve-78d464b6d8-snzsz\" (UID: \"371e82db-7c6c-411d-8f38-988f76bcf821\") " pod="e2e-models-e2e-worker-w0-36ccd3/premium-w0-36ccd3-kserve-78d464b6d8-snzsz" Nov 04 03:31:43.624802 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:31:43.624675 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/371e82db-7c6c-411d-8f38-988f76bcf821-model-cache\") pod \"premium-w0-36ccd3-kserve-78d464b6d8-snzsz\" (UID: \"371e82db-7c6c-411d-8f38-988f76bcf821\") " pod="e2e-models-e2e-worker-w0-36ccd3/premium-w0-36ccd3-kserve-78d464b6d8-snzsz" Nov 04 03:31:43.624844 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:31:43.624822 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/371e82db-7c6c-411d-8f38-988f76bcf821-home\") pod \"premium-w0-36ccd3-kserve-78d464b6d8-snzsz\" (UID: \"371e82db-7c6c-411d-8f38-988f76bcf821\") " pod="e2e-models-e2e-worker-w0-36ccd3/premium-w0-36ccd3-kserve-78d464b6d8-snzsz" Nov 04 03:31:43.624887 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:31:43.624871 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/371e82db-7c6c-411d-8f38-988f76bcf821-tmp-dir\") pod \"premium-w0-36ccd3-kserve-78d464b6d8-snzsz\" (UID: \"371e82db-7c6c-411d-8f38-988f76bcf821\") " pod="e2e-models-e2e-worker-w0-36ccd3/premium-w0-36ccd3-kserve-78d464b6d8-snzsz" Nov 04 03:31:43.624934 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:31:43.624921 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/371e82db-7c6c-411d-8f38-988f76bcf821-tls-certs\") pod \"premium-w0-36ccd3-kserve-78d464b6d8-snzsz\" (UID: \"371e82db-7c6c-411d-8f38-988f76bcf821\") " pod="e2e-models-e2e-worker-w0-36ccd3/premium-w0-36ccd3-kserve-78d464b6d8-snzsz" Nov 04 03:31:43.625012 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:31:43.624996 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtqtf\" (UniqueName: \"kubernetes.io/projected/371e82db-7c6c-411d-8f38-988f76bcf821-kube-api-access-rtqtf\") pod \"premium-w0-36ccd3-kserve-78d464b6d8-snzsz\" (UID: \"371e82db-7c6c-411d-8f38-988f76bcf821\") " pod="e2e-models-e2e-worker-w0-36ccd3/premium-w0-36ccd3-kserve-78d464b6d8-snzsz" Nov 04 03:31:43.625083 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:31:43.625058 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/371e82db-7c6c-411d-8f38-988f76bcf821-model-cache\") pod \"premium-w0-36ccd3-kserve-78d464b6d8-snzsz\" (UID: \"371e82db-7c6c-411d-8f38-988f76bcf821\") " pod="e2e-models-e2e-worker-w0-36ccd3/premium-w0-36ccd3-kserve-78d464b6d8-snzsz" Nov 04 03:31:43.625217 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:31:43.625193 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/371e82db-7c6c-411d-8f38-988f76bcf821-home\") pod \"premium-w0-36ccd3-kserve-78d464b6d8-snzsz\" (UID: \"371e82db-7c6c-411d-8f38-988f76bcf821\") " pod="e2e-models-e2e-worker-w0-36ccd3/premium-w0-36ccd3-kserve-78d464b6d8-snzsz" Nov 04 03:31:43.625292 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:31:43.625201 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/371e82db-7c6c-411d-8f38-988f76bcf821-tmp-dir\") pod \"premium-w0-36ccd3-kserve-78d464b6d8-snzsz\" (UID: \"371e82db-7c6c-411d-8f38-988f76bcf821\") " pod="e2e-models-e2e-worker-w0-36ccd3/premium-w0-36ccd3-kserve-78d464b6d8-snzsz" Nov 04 03:31:43.627108 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:31:43.627088 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/371e82db-7c6c-411d-8f38-988f76bcf821-dshm\") pod \"premium-w0-36ccd3-kserve-78d464b6d8-snzsz\" (UID: \"371e82db-7c6c-411d-8f38-988f76bcf821\") " pod="e2e-models-e2e-worker-w0-36ccd3/premium-w0-36ccd3-kserve-78d464b6d8-snzsz" Nov 04 03:31:43.627456 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:31:43.627436 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/371e82db-7c6c-411d-8f38-988f76bcf821-tls-certs\") pod \"premium-w0-36ccd3-kserve-78d464b6d8-snzsz\" (UID: \"371e82db-7c6c-411d-8f38-988f76bcf821\") " pod="e2e-models-e2e-worker-w0-36ccd3/premium-w0-36ccd3-kserve-78d464b6d8-snzsz" Nov 04 03:31:43.686593 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:31:43.686517 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtqtf\" (UniqueName: \"kubernetes.io/projected/371e82db-7c6c-411d-8f38-988f76bcf821-kube-api-access-rtqtf\") pod \"premium-w0-36ccd3-kserve-78d464b6d8-snzsz\" (UID: \"371e82db-7c6c-411d-8f38-988f76bcf821\") " pod="e2e-models-e2e-worker-w0-36ccd3/premium-w0-36ccd3-kserve-78d464b6d8-snzsz" Nov 04 03:31:43.705481 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:31:43.705442 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-36ccd3/premium-w0-36ccd3-kserve-78d464b6d8-snzsz" Nov 04 03:31:44.081537 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:31:44.081514 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-36ccd3/premium-w0-36ccd3-kserve-78d464b6d8-snzsz"] Nov 04 03:31:44.084357 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:31:44.084332 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod371e82db_7c6c_411d_8f38_988f76bcf821.slice/crio-df6888dbb06445ce3f7b4d13ef41b4b9bdd30551acfa39607e967aec52c88a02 WatchSource:0}: Error finding container df6888dbb06445ce3f7b4d13ef41b4b9bdd30551acfa39607e967aec52c88a02: Status 404 returned error can't find the container with id df6888dbb06445ce3f7b4d13ef41b4b9bdd30551acfa39607e967aec52c88a02 Nov 04 03:31:44.102387 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:31:44.102357 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-36ccd3/premium-w0-36ccd3-kserve-78d464b6d8-snzsz" event={"ID":"371e82db-7c6c-411d-8f38-988f76bcf821","Type":"ContainerStarted","Data":"df6888dbb06445ce3f7b4d13ef41b4b9bdd30551acfa39607e967aec52c88a02"} Nov 04 03:31:45.110890 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:31:45.110856 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-36ccd3/premium-w0-36ccd3-kserve-78d464b6d8-snzsz" event={"ID":"371e82db-7c6c-411d-8f38-988f76bcf821","Type":"ContainerStarted","Data":"39d094f1e15705a5b46354883de04c8c4bb49040f78603a3478119bae7af6308"} Nov 04 03:31:45.230430 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:31:45.230375 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-36ccd3/premium-w0-36ccd3-kserve-78d464b6d8-snzsz" podStartSLOduration=2.230360059 podStartE2EDuration="2.230360059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:31:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:31:45.229366199 +0000 UTC m=+830.503996962" watchObservedRunningTime="2026-11-04 03:31:45.230360059 +0000 UTC m=+830.504990821" Nov 04 03:31:45.705680 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:31:45.705648 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-36ccd3/premium-w0-36ccd3-kserve-78d464b6d8-snzsz" Nov 04 03:31:46.384341 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:31:46.384304 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:31:53.706307 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:31:53.706272 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-36ccd3/premium-w0-36ccd3-kserve-78d464b6d8-snzsz" Nov 04 03:31:53.718714 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:31:53.718682 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-36ccd3/premium-w0-36ccd3-kserve-78d464b6d8-snzsz" Nov 04 03:31:54.178972 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:31:54.178945 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-36ccd3/premium-w0-36ccd3-kserve-78d464b6d8-snzsz" Nov 04 03:32:16.386359 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:32:16.386309 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:32:20.895869 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:20.895830 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg"] Nov 04 03:32:20.896391 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:20.896039 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" podUID="0122d333-f982-4664-bb39-62f707a44370" containerName="istio-proxy" containerID="cri-o://bbb2bdad1085894658fb18e0a23ae75abadbaee44573bd537abd35e8a201a399" gracePeriod=30 Nov 04 03:32:20.946082 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:20.946050 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4"] Nov 04 03:32:20.946287 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:20.946265 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" podUID="162aa7e4-1372-460b-b6eb-2d12fbb5ea51" containerName="istio-proxy" containerID="cri-o://a214581bfc771d5977c465decc892e6281e5011dc1f2c1112a04ba6a7cb094ef" gracePeriod=30 Nov 04 03:32:26.207823 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.207796 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" Nov 04 03:32:26.237245 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.237221 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" Nov 04 03:32:26.313248 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.313219 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-workload-socket\") pod \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\" (UID: \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\") " Nov 04 03:32:26.313248 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.313253 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-istio-envoy\") pod \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\" (UID: \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\") " Nov 04 03:32:26.313499 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.313291 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-kube-api-access-twswk\" (UniqueName: \"kubernetes.io/projected/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-kube-api-access-twswk\") pod \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\" (UID: \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\") " Nov 04 03:32:26.313499 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.313326 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-istio-podinfo\") pod \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\" (UID: \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\") " Nov 04 03:32:26.313499 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.313398 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-istiod-ca-cert\") pod \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\" (UID: \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\") " Nov 04 03:32:26.313499 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.313433 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0122d333-f982-4664-bb39-62f707a44370-kube-api-access-r25gs\" (UniqueName: \"kubernetes.io/projected/0122d333-f982-4664-bb39-62f707a44370-kube-api-access-r25gs\") pod \"0122d333-f982-4664-bb39-62f707a44370\" (UID: \"0122d333-f982-4664-bb39-62f707a44370\") " Nov 04 03:32:26.313499 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.313483 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0122d333-f982-4664-bb39-62f707a44370-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0122d333-f982-4664-bb39-62f707a44370-istio-data\") pod \"0122d333-f982-4664-bb39-62f707a44370\" (UID: \"0122d333-f982-4664-bb39-62f707a44370\") " Nov 04 03:32:26.313779 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.313517 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-workload-socket" pod "162aa7e4-1372-460b-b6eb-2d12fbb5ea51" (UID: "162aa7e4-1372-460b-b6eb-2d12fbb5ea51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:32:26.313779 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.313558 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0122d333-f982-4664-bb39-62f707a44370-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0122d333-f982-4664-bb39-62f707a44370-istiod-ca-cert\") pod \"0122d333-f982-4664-bb39-62f707a44370\" (UID: \"0122d333-f982-4664-bb39-62f707a44370\") " Nov 04 03:32:26.313779 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.313594 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0122d333-f982-4664-bb39-62f707a44370-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0122d333-f982-4664-bb39-62f707a44370-istio-podinfo\") pod \"0122d333-f982-4664-bb39-62f707a44370\" (UID: \"0122d333-f982-4664-bb39-62f707a44370\") " Nov 04 03:32:26.313779 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.313710 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-istio-data\") pod \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\" (UID: \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\") " Nov 04 03:32:26.313779 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.313733 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0122d333-f982-4664-bb39-62f707a44370-istio-data" pod "0122d333-f982-4664-bb39-62f707a44370" (UID: "0122d333-f982-4664-bb39-62f707a44370"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:32:26.313779 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.313765 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-workload-certs\") pod \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\" (UID: \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\") " Nov 04 03:32:26.314084 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.313809 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0122d333-f982-4664-bb39-62f707a44370-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0122d333-f982-4664-bb39-62f707a44370-istio-envoy\") pod \"0122d333-f982-4664-bb39-62f707a44370\" (UID: \"0122d333-f982-4664-bb39-62f707a44370\") " Nov 04 03:32:26.314084 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.313845 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-istio-token\" (UniqueName: \"kubernetes.io/projected/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-istio-token\") pod \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\" (UID: \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\") " Nov 04 03:32:26.314084 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.313890 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0122d333-f982-4664-bb39-62f707a44370-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0122d333-f982-4664-bb39-62f707a44370-workload-certs\") pod \"0122d333-f982-4664-bb39-62f707a44370\" (UID: \"0122d333-f982-4664-bb39-62f707a44370\") " Nov 04 03:32:26.314084 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.313915 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-istiod-ca-cert" pod "162aa7e4-1372-460b-b6eb-2d12fbb5ea51" (UID: "162aa7e4-1372-460b-b6eb-2d12fbb5ea51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:32:26.314084 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.313948 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-credential-socket\") pod \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\" (UID: \"162aa7e4-1372-460b-b6eb-2d12fbb5ea51\") " Nov 04 03:32:26.314084 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.313983 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0122d333-f982-4664-bb39-62f707a44370-istiod-ca-cert" pod "0122d333-f982-4664-bb39-62f707a44370" (UID: "0122d333-f982-4664-bb39-62f707a44370"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:32:26.314084 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.313986 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0122d333-f982-4664-bb39-62f707a44370-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0122d333-f982-4664-bb39-62f707a44370-credential-socket\") pod \"0122d333-f982-4664-bb39-62f707a44370\" (UID: \"0122d333-f982-4664-bb39-62f707a44370\") " Nov 04 03:32:26.314084 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.314050 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0122d333-f982-4664-bb39-62f707a44370-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0122d333-f982-4664-bb39-62f707a44370-workload-socket\") pod \"0122d333-f982-4664-bb39-62f707a44370\" (UID: \"0122d333-f982-4664-bb39-62f707a44370\") " Nov 04 03:32:26.314701 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.314190 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0122d333-f982-4664-bb39-62f707a44370-credential-socket" pod "0122d333-f982-4664-bb39-62f707a44370" (UID: "0122d333-f982-4664-bb39-62f707a44370"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:32:26.314701 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.314118 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0122d333-f982-4664-bb39-62f707a44370-istio-token\" (UniqueName: \"kubernetes.io/projected/0122d333-f982-4664-bb39-62f707a44370-istio-token\") pod \"0122d333-f982-4664-bb39-62f707a44370\" (UID: \"0122d333-f982-4664-bb39-62f707a44370\") " Nov 04 03:32:26.314883 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.314860 5953 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0122d333-f982-4664-bb39-62f707a44370-credential-socket\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:32:26.315069 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.315054 5953 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-workload-socket\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:32:26.315230 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.315209 5953 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-istiod-ca-cert\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:32:26.315340 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.315235 5953 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0122d333-f982-4664-bb39-62f707a44370-istio-data\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:32:26.315340 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.315250 5953 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0122d333-f982-4664-bb39-62f707a44370-istiod-ca-cert\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:32:26.315340 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.314876 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0122d333-f982-4664-bb39-62f707a44370-workload-socket" pod "0122d333-f982-4664-bb39-62f707a44370" (UID: "0122d333-f982-4664-bb39-62f707a44370"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:32:26.315340 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.315069 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-credential-socket" pod "162aa7e4-1372-460b-b6eb-2d12fbb5ea51" (UID: "162aa7e4-1372-460b-b6eb-2d12fbb5ea51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:32:26.315340 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.315082 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0122d333-f982-4664-bb39-62f707a44370-workload-certs" pod "0122d333-f982-4664-bb39-62f707a44370" (UID: "0122d333-f982-4664-bb39-62f707a44370"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:32:26.315340 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.315299 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-workload-certs" pod "162aa7e4-1372-460b-b6eb-2d12fbb5ea51" (UID: "162aa7e4-1372-460b-b6eb-2d12fbb5ea51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:32:26.315774 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.315488 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-istio-data" pod "162aa7e4-1372-460b-b6eb-2d12fbb5ea51" (UID: "162aa7e4-1372-460b-b6eb-2d12fbb5ea51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:32:26.316332 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.316285 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-kube-api-access-twswk" pod "162aa7e4-1372-460b-b6eb-2d12fbb5ea51" (UID: "162aa7e4-1372-460b-b6eb-2d12fbb5ea51"). InnerVolumeSpecName "kube-api-access-twswk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:32:26.316561 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.316535 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0122d333-f982-4664-bb39-62f707a44370-istio-podinfo" pod "0122d333-f982-4664-bb39-62f707a44370" (UID: "0122d333-f982-4664-bb39-62f707a44370"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:32:26.316682 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.316568 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-istio-podinfo" pod "162aa7e4-1372-460b-b6eb-2d12fbb5ea51" (UID: "162aa7e4-1372-460b-b6eb-2d12fbb5ea51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:32:26.317051 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.317026 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-istio-token" pod "162aa7e4-1372-460b-b6eb-2d12fbb5ea51" (UID: "162aa7e4-1372-460b-b6eb-2d12fbb5ea51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:32:26.317493 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.317470 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0122d333-f982-4664-bb39-62f707a44370-istio-token" pod "0122d333-f982-4664-bb39-62f707a44370" (UID: "0122d333-f982-4664-bb39-62f707a44370"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:32:26.317543 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.317527 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0122d333-f982-4664-bb39-62f707a44370-kube-api-access-r25gs" pod "0122d333-f982-4664-bb39-62f707a44370" (UID: "0122d333-f982-4664-bb39-62f707a44370"). InnerVolumeSpecName "kube-api-access-r25gs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:32:26.317664 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.317643 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-istio-envoy" pod "162aa7e4-1372-460b-b6eb-2d12fbb5ea51" (UID: "162aa7e4-1372-460b-b6eb-2d12fbb5ea51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:32:26.317753 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.317739 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0122d333-f982-4664-bb39-62f707a44370-istio-envoy" pod "0122d333-f982-4664-bb39-62f707a44370" (UID: "0122d333-f982-4664-bb39-62f707a44370"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:32:26.377212 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.377178 5953 generic.go:356] "Generic (PLEG): container finished" podID="162aa7e4-1372-460b-b6eb-2d12fbb5ea51" containerID="a214581bfc771d5977c465decc892e6281e5011dc1f2c1112a04ba6a7cb094ef" exitCode=0 Nov 04 03:32:26.377417 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.377239 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" Nov 04 03:32:26.377417 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.377271 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" event={"ID":"162aa7e4-1372-460b-b6eb-2d12fbb5ea51","Type":"ContainerDied","Data":"a214581bfc771d5977c465decc892e6281e5011dc1f2c1112a04ba6a7cb094ef"} Nov 04 03:32:26.377417 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.377331 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4" event={"ID":"162aa7e4-1372-460b-b6eb-2d12fbb5ea51","Type":"ContainerDied","Data":"9c827252ddc0a707a102846f4ca61e49869b73afe375a131bef02d48fad26517"} Nov 04 03:32:26.377417 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.377360 5953 scope.go:122] "RemoveContainer" containerID="a214581bfc771d5977c465decc892e6281e5011dc1f2c1112a04ba6a7cb094ef" Nov 04 03:32:26.378902 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.378875 5953 generic.go:356] "Generic (PLEG): container finished" podID="0122d333-f982-4664-bb39-62f707a44370" containerID="bbb2bdad1085894658fb18e0a23ae75abadbaee44573bd537abd35e8a201a399" exitCode=0 Nov 04 03:32:26.379044 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.378931 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" Nov 04 03:32:26.379111 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.379045 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" event={"ID":"0122d333-f982-4664-bb39-62f707a44370","Type":"ContainerDied","Data":"bbb2bdad1085894658fb18e0a23ae75abadbaee44573bd537abd35e8a201a399"} Nov 04 03:32:26.379167 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.379107 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg" event={"ID":"0122d333-f982-4664-bb39-62f707a44370","Type":"ContainerDied","Data":"fd4eb46367cb6c42de06f2fc32bde782023533a14c6df70b5a2eb5fe85911882"} Nov 04 03:32:26.396442 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.396418 5953 scope.go:122] "RemoveContainer" containerID="a214581bfc771d5977c465decc892e6281e5011dc1f2c1112a04ba6a7cb094ef" Nov 04 03:32:26.396826 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:32:26.396807 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a214581bfc771d5977c465decc892e6281e5011dc1f2c1112a04ba6a7cb094ef\": container with ID starting with a214581bfc771d5977c465decc892e6281e5011dc1f2c1112a04ba6a7cb094ef not found: ID does not exist" containerID="a214581bfc771d5977c465decc892e6281e5011dc1f2c1112a04ba6a7cb094ef" Nov 04 03:32:26.396904 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.396835 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a214581bfc771d5977c465decc892e6281e5011dc1f2c1112a04ba6a7cb094ef"} err="failed to get container status \"a214581bfc771d5977c465decc892e6281e5011dc1f2c1112a04ba6a7cb094ef\": rpc error: code = NotFound desc = could not find container \"a214581bfc771d5977c465decc892e6281e5011dc1f2c1112a04ba6a7cb094ef\": container with ID starting with a214581bfc771d5977c465decc892e6281e5011dc1f2c1112a04ba6a7cb094ef not found: ID does not exist" Nov 04 03:32:26.396904 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.396854 5953 scope.go:122] "RemoveContainer" containerID="bbb2bdad1085894658fb18e0a23ae75abadbaee44573bd537abd35e8a201a399" Nov 04 03:32:26.407419 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.407385 5953 scope.go:122] "RemoveContainer" containerID="bbb2bdad1085894658fb18e0a23ae75abadbaee44573bd537abd35e8a201a399" Nov 04 03:32:26.407862 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:32:26.407838 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbb2bdad1085894658fb18e0a23ae75abadbaee44573bd537abd35e8a201a399\": container with ID starting with bbb2bdad1085894658fb18e0a23ae75abadbaee44573bd537abd35e8a201a399 not found: ID does not exist" containerID="bbb2bdad1085894658fb18e0a23ae75abadbaee44573bd537abd35e8a201a399" Nov 04 03:32:26.407965 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.407886 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbb2bdad1085894658fb18e0a23ae75abadbaee44573bd537abd35e8a201a399"} err="failed to get container status \"bbb2bdad1085894658fb18e0a23ae75abadbaee44573bd537abd35e8a201a399\": rpc error: code = NotFound desc = could not find container \"bbb2bdad1085894658fb18e0a23ae75abadbaee44573bd537abd35e8a201a399\": container with ID starting with bbb2bdad1085894658fb18e0a23ae75abadbaee44573bd537abd35e8a201a399 not found: ID does not exist" Nov 04 03:32:26.416756 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.416692 5953 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0122d333-f982-4664-bb39-62f707a44370-istio-token\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:32:26.416756 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.416724 5953 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-istio-envoy\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:32:26.416756 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.416741 5953 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twswk\" (UniqueName: \"kubernetes.io/projected/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-kube-api-access-twswk\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:32:26.416756 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.416754 5953 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-istio-podinfo\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:32:26.416960 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.416770 5953 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r25gs\" (UniqueName: \"kubernetes.io/projected/0122d333-f982-4664-bb39-62f707a44370-kube-api-access-r25gs\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:32:26.416960 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.416784 5953 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0122d333-f982-4664-bb39-62f707a44370-istio-podinfo\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:32:26.416960 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.416797 5953 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-istio-data\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:32:26.416960 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.416812 5953 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-workload-certs\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:32:26.416960 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.416825 5953 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0122d333-f982-4664-bb39-62f707a44370-istio-envoy\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:32:26.416960 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.416838 5953 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-istio-token\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:32:26.416960 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.416853 5953 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0122d333-f982-4664-bb39-62f707a44370-workload-certs\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:32:26.416960 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.416867 5953 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/162aa7e4-1372-460b-b6eb-2d12fbb5ea51-credential-socket\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:32:26.416960 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.416883 5953 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0122d333-f982-4664-bb39-62f707a44370-workload-socket\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:32:26.670221 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.670137 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg"] Nov 04 03:32:26.676235 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.676207 5953 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdxg"] Nov 04 03:32:26.786365 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.786331 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4"] Nov 04 03:32:26.789737 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.789713 5953 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vtsx4"] Nov 04 03:32:26.855577 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.855547 5953 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0122d333-f982-4664-bb39-62f707a44370" path="/var/lib/kubelet/pods/0122d333-f982-4664-bb39-62f707a44370/volumes" Nov 04 03:32:26.856002 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:26.855989 5953 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="162aa7e4-1372-460b-b6eb-2d12fbb5ea51" path="/var/lib/kubelet/pods/162aa7e4-1372-460b-b6eb-2d12fbb5ea51/volumes" Nov 04 03:32:46.386153 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:32:46.386115 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:32:57.091757 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:57.091720 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27"] Nov 04 03:32:57.092387 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:57.092370 5953 cpu_manager.go:414] "containerMap: removing container" podUID="162aa7e4-1372-460b-b6eb-2d12fbb5ea51" containerName="istio-proxy" Nov 04 03:32:57.092434 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:57.092390 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="162aa7e4-1372-460b-b6eb-2d12fbb5ea51" containerName="istio-proxy" Nov 04 03:32:57.092434 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:57.092417 5953 cpu_manager.go:414] "containerMap: removing container" podUID="0122d333-f982-4664-bb39-62f707a44370" containerName="istio-proxy" Nov 04 03:32:57.092434 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:57.092422 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0122d333-f982-4664-bb39-62f707a44370" containerName="istio-proxy" Nov 04 03:32:57.092525 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:57.092514 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="0122d333-f982-4664-bb39-62f707a44370" containerName="istio-proxy" Nov 04 03:32:57.092558 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:57.092526 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="162aa7e4-1372-460b-b6eb-2d12fbb5ea51" containerName="istio-proxy" Nov 04 03:32:57.097302 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:57.097276 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27" Nov 04 03:32:57.154015 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:57.153980 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27"] Nov 04 03:32:57.162795 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:57.162762 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w2sb\" (UniqueName: \"kubernetes.io/projected/40a71a57-c1dd-4119-83b9-22527ca0364f-kube-api-access-2w2sb\") pod \"distinct-w3-c5bec0-kserve-769854457b-mxs27\" (UID: \"40a71a57-c1dd-4119-83b9-22527ca0364f\") " pod="e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27" Nov 04 03:32:57.162969 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:57.162806 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40a71a57-c1dd-4119-83b9-22527ca0364f-home\") pod \"distinct-w3-c5bec0-kserve-769854457b-mxs27\" (UID: \"40a71a57-c1dd-4119-83b9-22527ca0364f\") " pod="e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27" Nov 04 03:32:57.162969 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:57.162834 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40a71a57-c1dd-4119-83b9-22527ca0364f-tmp-dir\") pod \"distinct-w3-c5bec0-kserve-769854457b-mxs27\" (UID: \"40a71a57-c1dd-4119-83b9-22527ca0364f\") " pod="e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27" Nov 04 03:32:57.162969 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:57.162965 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40a71a57-c1dd-4119-83b9-22527ca0364f-model-cache\") pod \"distinct-w3-c5bec0-kserve-769854457b-mxs27\" (UID: \"40a71a57-c1dd-4119-83b9-22527ca0364f\") " pod="e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27" Nov 04 03:32:57.163142 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:57.163026 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40a71a57-c1dd-4119-83b9-22527ca0364f-dshm\") pod \"distinct-w3-c5bec0-kserve-769854457b-mxs27\" (UID: \"40a71a57-c1dd-4119-83b9-22527ca0364f\") " pod="e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27" Nov 04 03:32:57.163142 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:57.163091 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40a71a57-c1dd-4119-83b9-22527ca0364f-tls-certs\") pod \"distinct-w3-c5bec0-kserve-769854457b-mxs27\" (UID: \"40a71a57-c1dd-4119-83b9-22527ca0364f\") " pod="e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27" Nov 04 03:32:57.201738 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:57.201707 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-c5bec0\"/\"openshift-service-ca.crt\"" Nov 04 03:32:57.201940 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:57.201842 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-c5bec0\"/\"default-dockercfg-s2smx\"" Nov 04 03:32:57.201940 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:57.201868 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-c5bec0\"/\"distinct-w3-c5bec0-kserve-self-signed-certs\"" Nov 04 03:32:57.201940 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:57.201894 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-c5bec0\"/\"kube-root-ca.crt\"" Nov 04 03:32:57.264648 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:57.264589 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40a71a57-c1dd-4119-83b9-22527ca0364f-dshm\") pod \"distinct-w3-c5bec0-kserve-769854457b-mxs27\" (UID: \"40a71a57-c1dd-4119-83b9-22527ca0364f\") " pod="e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27" Nov 04 03:32:57.264648 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:57.264650 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40a71a57-c1dd-4119-83b9-22527ca0364f-tls-certs\") pod \"distinct-w3-c5bec0-kserve-769854457b-mxs27\" (UID: \"40a71a57-c1dd-4119-83b9-22527ca0364f\") " pod="e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27" Nov 04 03:32:57.264889 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:57.264706 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w2sb\" (UniqueName: \"kubernetes.io/projected/40a71a57-c1dd-4119-83b9-22527ca0364f-kube-api-access-2w2sb\") pod \"distinct-w3-c5bec0-kserve-769854457b-mxs27\" (UID: \"40a71a57-c1dd-4119-83b9-22527ca0364f\") " pod="e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27" Nov 04 03:32:57.264889 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:57.264734 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40a71a57-c1dd-4119-83b9-22527ca0364f-home\") pod \"distinct-w3-c5bec0-kserve-769854457b-mxs27\" (UID: \"40a71a57-c1dd-4119-83b9-22527ca0364f\") " pod="e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27" Nov 04 03:32:57.264889 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:57.264769 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40a71a57-c1dd-4119-83b9-22527ca0364f-tmp-dir\") pod \"distinct-w3-c5bec0-kserve-769854457b-mxs27\" (UID: \"40a71a57-c1dd-4119-83b9-22527ca0364f\") " pod="e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27" Nov 04 03:32:57.264889 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:57.264817 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40a71a57-c1dd-4119-83b9-22527ca0364f-model-cache\") pod \"distinct-w3-c5bec0-kserve-769854457b-mxs27\" (UID: \"40a71a57-c1dd-4119-83b9-22527ca0364f\") " pod="e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27" Nov 04 03:32:57.265154 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:57.265131 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40a71a57-c1dd-4119-83b9-22527ca0364f-home\") pod \"distinct-w3-c5bec0-kserve-769854457b-mxs27\" (UID: \"40a71a57-c1dd-4119-83b9-22527ca0364f\") " pod="e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27" Nov 04 03:32:57.265218 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:57.265158 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40a71a57-c1dd-4119-83b9-22527ca0364f-tmp-dir\") pod \"distinct-w3-c5bec0-kserve-769854457b-mxs27\" (UID: \"40a71a57-c1dd-4119-83b9-22527ca0364f\") " pod="e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27" Nov 04 03:32:57.265277 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:57.265212 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40a71a57-c1dd-4119-83b9-22527ca0364f-model-cache\") pod \"distinct-w3-c5bec0-kserve-769854457b-mxs27\" (UID: \"40a71a57-c1dd-4119-83b9-22527ca0364f\") " pod="e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27" Nov 04 03:32:57.267087 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:57.267062 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40a71a57-c1dd-4119-83b9-22527ca0364f-dshm\") pod \"distinct-w3-c5bec0-kserve-769854457b-mxs27\" (UID: \"40a71a57-c1dd-4119-83b9-22527ca0364f\") " pod="e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27" Nov 04 03:32:57.267359 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:57.267343 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40a71a57-c1dd-4119-83b9-22527ca0364f-tls-certs\") pod \"distinct-w3-c5bec0-kserve-769854457b-mxs27\" (UID: \"40a71a57-c1dd-4119-83b9-22527ca0364f\") " pod="e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27" Nov 04 03:32:57.326468 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:57.326432 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w2sb\" (UniqueName: \"kubernetes.io/projected/40a71a57-c1dd-4119-83b9-22527ca0364f-kube-api-access-2w2sb\") pod \"distinct-w3-c5bec0-kserve-769854457b-mxs27\" (UID: \"40a71a57-c1dd-4119-83b9-22527ca0364f\") " pod="e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27" Nov 04 03:32:57.423186 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:57.422852 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27" Nov 04 03:32:57.815869 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:57.815846 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27"] Nov 04 03:32:57.818254 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:32:57.818222 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40a71a57_c1dd_4119_83b9_22527ca0364f.slice/crio-4ee3b1effb924e20724aa27cf2bbc0b1b39d8ae3e0c1c0a9491efae27ae09990 WatchSource:0}: Error finding container 4ee3b1effb924e20724aa27cf2bbc0b1b39d8ae3e0c1c0a9491efae27ae09990: Status 404 returned error can't find the container with id 4ee3b1effb924e20724aa27cf2bbc0b1b39d8ae3e0c1c0a9491efae27ae09990 Nov 04 03:32:58.566150 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:58.566106 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27" event={"ID":"40a71a57-c1dd-4119-83b9-22527ca0364f","Type":"ContainerStarted","Data":"969533b92a44907f486f97cd11f223d10941daaefde41689cb61e2d43b53b398"} Nov 04 03:32:58.566150 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:58.566155 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27" event={"ID":"40a71a57-c1dd-4119-83b9-22527ca0364f","Type":"ContainerStarted","Data":"4ee3b1effb924e20724aa27cf2bbc0b1b39d8ae3e0c1c0a9491efae27ae09990"} Nov 04 03:32:58.688990 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:58.688745 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27" podStartSLOduration=1.688727773 podStartE2EDuration="1.688727773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:32:58.687155013 +0000 UTC m=+903.961785775" watchObservedRunningTime="2026-11-04 03:32:58.688727773 +0000 UTC m=+903.963358535" Nov 04 03:32:59.423532 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:32:59.423500 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27" Nov 04 03:33:07.424695 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:07.424028 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27" Nov 04 03:33:07.441791 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:07.441754 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27" Nov 04 03:33:07.627107 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:07.627074 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27" Nov 04 03:33:16.386042 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:33:16.385997 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:33:20.087210 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:20.087177 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7379-d99e5f"] Nov 04 03:33:20.091052 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:20.091034 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7379-d99e5f" Nov 04 03:33:20.143372 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:20.143340 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7388-12987b"] Nov 04 03:33:20.147636 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:20.147589 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7379-d99e5f"] Nov 04 03:33:20.147805 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:20.147721 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7388-12987b" Nov 04 03:33:20.192015 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:20.191975 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lhr5q\"" Nov 04 03:33:20.203579 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:20.203546 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7388-12987b"] Nov 04 03:33:20.215282 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:20.215233 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dv8j\" (UniqueName: \"kubernetes.io/projected/a4826b6b-4ae9-4947-891b-eb8dd25306cc-kube-api-access-4dv8j\") pod \"test-curl-7379-d99e5f\" (UID: \"a4826b6b-4ae9-4947-891b-eb8dd25306cc\") " pod="opendatahub/test-curl-7379-d99e5f" Nov 04 03:33:20.316867 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:20.316832 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvpvn\" (UniqueName: \"kubernetes.io/projected/a5bd7575-de37-42e3-8b0a-75a5be69eae8-kube-api-access-rvpvn\") pod \"test-curl-7388-12987b\" (UID: \"a5bd7575-de37-42e3-8b0a-75a5be69eae8\") " pod="opendatahub/test-curl-7388-12987b" Nov 04 03:33:20.317070 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:20.316894 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dv8j\" (UniqueName: \"kubernetes.io/projected/a4826b6b-4ae9-4947-891b-eb8dd25306cc-kube-api-access-4dv8j\") pod \"test-curl-7379-d99e5f\" (UID: \"a4826b6b-4ae9-4947-891b-eb8dd25306cc\") " pod="opendatahub/test-curl-7379-d99e5f" Nov 04 03:33:20.381237 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:20.381160 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dv8j\" (UniqueName: \"kubernetes.io/projected/a4826b6b-4ae9-4947-891b-eb8dd25306cc-kube-api-access-4dv8j\") pod \"test-curl-7379-d99e5f\" (UID: \"a4826b6b-4ae9-4947-891b-eb8dd25306cc\") " pod="opendatahub/test-curl-7379-d99e5f" Nov 04 03:33:20.403730 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:20.403695 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7379-d99e5f" Nov 04 03:33:20.417975 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:20.417941 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvpvn\" (UniqueName: \"kubernetes.io/projected/a5bd7575-de37-42e3-8b0a-75a5be69eae8-kube-api-access-rvpvn\") pod \"test-curl-7388-12987b\" (UID: \"a5bd7575-de37-42e3-8b0a-75a5be69eae8\") " pod="opendatahub/test-curl-7388-12987b" Nov 04 03:33:20.480375 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:20.480340 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvpvn\" (UniqueName: \"kubernetes.io/projected/a5bd7575-de37-42e3-8b0a-75a5be69eae8-kube-api-access-rvpvn\") pod \"test-curl-7388-12987b\" (UID: \"a5bd7575-de37-42e3-8b0a-75a5be69eae8\") " pod="opendatahub/test-curl-7388-12987b" Nov 04 03:33:20.760895 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:20.760857 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7388-12987b" Nov 04 03:33:20.778779 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:20.778755 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7379-d99e5f"] Nov 04 03:33:20.781063 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:33:20.781031 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4826b6b_4ae9_4947_891b_eb8dd25306cc.slice/crio-5b782e4a024cb68300bd2c4c842cad0481c1759cc78aecae3745ff80b2dcebfc WatchSource:0}: Error finding container 5b782e4a024cb68300bd2c4c842cad0481c1759cc78aecae3745ff80b2dcebfc: Status 404 returned error can't find the container with id 5b782e4a024cb68300bd2c4c842cad0481c1759cc78aecae3745ff80b2dcebfc Nov 04 03:33:20.783465 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:20.783444 5953 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 03:33:21.137103 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:21.137043 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7388-12987b"] Nov 04 03:33:21.139515 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:33:21.139488 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5bd7575_de37_42e3_8b0a_75a5be69eae8.slice/crio-776e50c91c8b8c52765d99fe100556eaa990bdd203c857dabe6cd6be41b1bef2 WatchSource:0}: Error finding container 776e50c91c8b8c52765d99fe100556eaa990bdd203c857dabe6cd6be41b1bef2: Status 404 returned error can't find the container with id 776e50c91c8b8c52765d99fe100556eaa990bdd203c857dabe6cd6be41b1bef2 Nov 04 03:33:21.693210 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:21.693173 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7388-12987b" event={"ID":"a5bd7575-de37-42e3-8b0a-75a5be69eae8","Type":"ContainerStarted","Data":"776e50c91c8b8c52765d99fe100556eaa990bdd203c857dabe6cd6be41b1bef2"} Nov 04 03:33:21.694438 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:21.694411 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7379-d99e5f" event={"ID":"a4826b6b-4ae9-4947-891b-eb8dd25306cc","Type":"ContainerStarted","Data":"5b782e4a024cb68300bd2c4c842cad0481c1759cc78aecae3745ff80b2dcebfc"} Nov 04 03:33:23.708752 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:23.708640 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7379-d99e5f" event={"ID":"a4826b6b-4ae9-4947-891b-eb8dd25306cc","Type":"ContainerStarted","Data":"b4b58913af50a91108edb5aacb7a21220c42c6e1834f03974240c98cf4c7a356"} Nov 04 03:33:23.710586 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:23.710558 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7388-12987b" event={"ID":"a5bd7575-de37-42e3-8b0a-75a5be69eae8","Type":"ContainerStarted","Data":"55c743562a8857d34eba31751c6f32c2f0ae5561c6858e2e89ba03c381a58757"} Nov 04 03:33:23.824179 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:23.824125 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7379-d99e5f" podStartSLOduration=1.165812145 podStartE2EDuration="3.824110163s" totalImagesPullingTime="2.658298018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:33:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:33:23.822505172 +0000 UTC m=+929.097135933" watchObservedRunningTime="2026-11-04 03:33:23.824110163 +0000 UTC m=+929.098740926" Nov 04 03:33:23.937675 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:23.937606 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7388-12987b" podStartSLOduration=1.494864574 podStartE2EDuration="3.93759052s" totalImagesPullingTime="2.442725946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:33:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:33:23.935894871 +0000 UTC m=+929.210525635" watchObservedRunningTime="2026-11-04 03:33:23.93759052 +0000 UTC m=+929.212221282" Nov 04 03:33:25.287766 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:25.287728 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7379-d99e5f"] Nov 04 03:33:25.299646 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:25.298526 5953 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7379-d99e5f"] Nov 04 03:33:25.415559 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:25.415521 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7388-12987b"] Nov 04 03:33:25.419518 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:25.419482 5953 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7388-12987b"] Nov 04 03:33:25.721277 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:25.721192 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7379-d99e5f" podUID="a4826b6b-4ae9-4947-891b-eb8dd25306cc" containerName="test-curl-7379-d99e5f" containerID="cri-o://b4b58913af50a91108edb5aacb7a21220c42c6e1834f03974240c98cf4c7a356" gracePeriod=2 Nov 04 03:33:25.721277 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:25.721212 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7388-12987b" podUID="a5bd7575-de37-42e3-8b0a-75a5be69eae8" containerName="test-curl-7388-12987b" containerID="cri-o://55c743562a8857d34eba31751c6f32c2f0ae5561c6858e2e89ba03c381a58757" gracePeriod=2 Nov 04 03:33:25.773076 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:33:25.773042 5953 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7379-d99e5f\" is forbidden: User \"system:node:ip-10-0-133-6.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-6.us-west-1.compute.internal' and this object" podUID="a4826b6b-4ae9-4947-891b-eb8dd25306cc" pod="opendatahub/test-curl-7379-d99e5f" Nov 04 03:33:25.824982 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:33:25.824941 5953 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7388-12987b\" is forbidden: User \"system:node:ip-10-0-133-6.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-6.us-west-1.compute.internal' and this object" podUID="a5bd7575-de37-42e3-8b0a-75a5be69eae8" pod="opendatahub/test-curl-7388-12987b" Nov 04 03:33:25.876421 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:33:25.876385 5953 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7379-d99e5f\" is forbidden: User \"system:node:ip-10-0-133-6.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-6.us-west-1.compute.internal' and this object" podUID="a4826b6b-4ae9-4947-891b-eb8dd25306cc" pod="opendatahub/test-curl-7379-d99e5f" Nov 04 03:33:25.927927 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:33:25.927885 5953 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7388-12987b\" is forbidden: User \"system:node:ip-10-0-133-6.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-6.us-west-1.compute.internal' and this object" podUID="a5bd7575-de37-42e3-8b0a-75a5be69eae8" pod="opendatahub/test-curl-7388-12987b" Nov 04 03:33:27.511843 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.511807 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4"] Nov 04 03:33:27.512326 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.512313 5953 cpu_manager.go:414] "containerMap: removing container" podUID="a5bd7575-de37-42e3-8b0a-75a5be69eae8" containerName="test-curl-7388-12987b" Nov 04 03:33:27.512375 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.512327 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5bd7575-de37-42e3-8b0a-75a5be69eae8" containerName="test-curl-7388-12987b" Nov 04 03:33:27.512375 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.512360 5953 cpu_manager.go:414] "containerMap: removing container" podUID="a4826b6b-4ae9-4947-891b-eb8dd25306cc" containerName="test-curl-7379-d99e5f" Nov 04 03:33:27.512375 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.512365 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4826b6b-4ae9-4947-891b-eb8dd25306cc" containerName="test-curl-7379-d99e5f" Nov 04 03:33:27.512466 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.512445 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5bd7575-de37-42e3-8b0a-75a5be69eae8" containerName="test-curl-7388-12987b" Nov 04 03:33:27.512466 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.512458 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4826b6b-4ae9-4947-891b-eb8dd25306cc" containerName="test-curl-7379-d99e5f" Nov 04 03:33:27.515809 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.515793 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" Nov 04 03:33:27.568688 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.568648 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-141891-openshift-default-dockercfg-55ft9\"" Nov 04 03:33:27.598025 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.597993 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48e61829-0b7d-40a9-b877-398aa633b44f-istio-token\") pod \"e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4\" (UID: \"48e61829-0b7d-40a9-b877-398aa633b44f\") " pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" Nov 04 03:33:27.598217 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.598072 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pbsg\" (UniqueName: \"kubernetes.io/projected/48e61829-0b7d-40a9-b877-398aa633b44f-kube-api-access-9pbsg\") pod \"e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4\" (UID: \"48e61829-0b7d-40a9-b877-398aa633b44f\") " pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" Nov 04 03:33:27.598217 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.598121 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e61829-0b7d-40a9-b877-398aa633b44f-workload-socket\") pod \"e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4\" (UID: \"48e61829-0b7d-40a9-b877-398aa633b44f\") " pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" Nov 04 03:33:27.598217 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.598143 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48e61829-0b7d-40a9-b877-398aa633b44f-istio-podinfo\") pod \"e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4\" (UID: \"48e61829-0b7d-40a9-b877-398aa633b44f\") " pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" Nov 04 03:33:27.598370 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.598254 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48e61829-0b7d-40a9-b877-398aa633b44f-istiod-ca-cert\") pod \"e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4\" (UID: \"48e61829-0b7d-40a9-b877-398aa633b44f\") " pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" Nov 04 03:33:27.598370 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.598300 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e61829-0b7d-40a9-b877-398aa633b44f-credential-socket\") pod \"e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4\" (UID: \"48e61829-0b7d-40a9-b877-398aa633b44f\") " pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" Nov 04 03:33:27.598370 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.598355 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48e61829-0b7d-40a9-b877-398aa633b44f-istio-envoy\") pod \"e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4\" (UID: \"48e61829-0b7d-40a9-b877-398aa633b44f\") " pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" Nov 04 03:33:27.598468 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.598376 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48e61829-0b7d-40a9-b877-398aa633b44f-istio-data\") pod \"e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4\" (UID: \"48e61829-0b7d-40a9-b877-398aa633b44f\") " pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" Nov 04 03:33:27.598503 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.598476 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48e61829-0b7d-40a9-b877-398aa633b44f-workload-certs\") pod \"e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4\" (UID: \"48e61829-0b7d-40a9-b877-398aa633b44f\") " pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" Nov 04 03:33:27.625218 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.625185 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4"] Nov 04 03:33:27.699830 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.699797 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pbsg\" (UniqueName: \"kubernetes.io/projected/48e61829-0b7d-40a9-b877-398aa633b44f-kube-api-access-9pbsg\") pod \"e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4\" (UID: \"48e61829-0b7d-40a9-b877-398aa633b44f\") " pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" Nov 04 03:33:27.700015 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.699840 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e61829-0b7d-40a9-b877-398aa633b44f-workload-socket\") pod \"e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4\" (UID: \"48e61829-0b7d-40a9-b877-398aa633b44f\") " pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" Nov 04 03:33:27.700015 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.699856 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48e61829-0b7d-40a9-b877-398aa633b44f-istio-podinfo\") pod \"e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4\" (UID: \"48e61829-0b7d-40a9-b877-398aa633b44f\") " pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" Nov 04 03:33:27.700015 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.699907 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48e61829-0b7d-40a9-b877-398aa633b44f-istiod-ca-cert\") pod \"e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4\" (UID: \"48e61829-0b7d-40a9-b877-398aa633b44f\") " pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" Nov 04 03:33:27.700015 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.699943 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e61829-0b7d-40a9-b877-398aa633b44f-credential-socket\") pod \"e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4\" (UID: \"48e61829-0b7d-40a9-b877-398aa633b44f\") " pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" Nov 04 03:33:27.700015 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.699999 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48e61829-0b7d-40a9-b877-398aa633b44f-istio-envoy\") pod \"e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4\" (UID: \"48e61829-0b7d-40a9-b877-398aa633b44f\") " pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" Nov 04 03:33:27.700288 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.700043 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48e61829-0b7d-40a9-b877-398aa633b44f-istio-data\") pod \"e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4\" (UID: \"48e61829-0b7d-40a9-b877-398aa633b44f\") " pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" Nov 04 03:33:27.700288 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.700099 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48e61829-0b7d-40a9-b877-398aa633b44f-workload-certs\") pod \"e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4\" (UID: \"48e61829-0b7d-40a9-b877-398aa633b44f\") " pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" Nov 04 03:33:27.700288 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.700215 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48e61829-0b7d-40a9-b877-398aa633b44f-istio-token\") pod \"e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4\" (UID: \"48e61829-0b7d-40a9-b877-398aa633b44f\") " pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" Nov 04 03:33:27.700288 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.700272 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e61829-0b7d-40a9-b877-398aa633b44f-workload-socket\") pod \"e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4\" (UID: \"48e61829-0b7d-40a9-b877-398aa633b44f\") " pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" Nov 04 03:33:27.700510 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.700298 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e61829-0b7d-40a9-b877-398aa633b44f-credential-socket\") pod \"e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4\" (UID: \"48e61829-0b7d-40a9-b877-398aa633b44f\") " pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" Nov 04 03:33:27.700510 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.700481 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48e61829-0b7d-40a9-b877-398aa633b44f-istio-data\") pod \"e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4\" (UID: \"48e61829-0b7d-40a9-b877-398aa633b44f\") " pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" Nov 04 03:33:27.700658 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.700555 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48e61829-0b7d-40a9-b877-398aa633b44f-workload-certs\") pod \"e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4\" (UID: \"48e61829-0b7d-40a9-b877-398aa633b44f\") " pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" Nov 04 03:33:27.700780 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.700723 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48e61829-0b7d-40a9-b877-398aa633b44f-istiod-ca-cert\") pod \"e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4\" (UID: \"48e61829-0b7d-40a9-b877-398aa633b44f\") " pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" Nov 04 03:33:27.702746 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.702711 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48e61829-0b7d-40a9-b877-398aa633b44f-istio-envoy\") pod \"e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4\" (UID: \"48e61829-0b7d-40a9-b877-398aa633b44f\") " pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" Nov 04 03:33:27.702985 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.702961 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48e61829-0b7d-40a9-b877-398aa633b44f-istio-podinfo\") pod \"e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4\" (UID: \"48e61829-0b7d-40a9-b877-398aa633b44f\") " pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" Nov 04 03:33:27.765527 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.765034 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pbsg\" (UniqueName: \"kubernetes.io/projected/48e61829-0b7d-40a9-b877-398aa633b44f-kube-api-access-9pbsg\") pod \"e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4\" (UID: \"48e61829-0b7d-40a9-b877-398aa633b44f\") " pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" Nov 04 03:33:27.765695 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.765588 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48e61829-0b7d-40a9-b877-398aa633b44f-istio-token\") pod \"e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4\" (UID: \"48e61829-0b7d-40a9-b877-398aa633b44f\") " pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" Nov 04 03:33:27.829234 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.829180 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" Nov 04 03:33:27.845643 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.845600 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw"] Nov 04 03:33:27.849937 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.849900 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" Nov 04 03:33:27.903159 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.898361 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7388-12987b" Nov 04 03:33:27.903159 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.901507 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5bd7575-de37-42e3-8b0a-75a5be69eae8-kube-api-access-rvpvn\" (UniqueName: \"kubernetes.io/projected/a5bd7575-de37-42e3-8b0a-75a5be69eae8-kube-api-access-rvpvn\") pod \"a5bd7575-de37-42e3-8b0a-75a5be69eae8\" (UID: \"a5bd7575-de37-42e3-8b0a-75a5be69eae8\") " Nov 04 03:33:27.903159 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.901930 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-istio-data\") pod \"e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw\" (UID: \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\") " pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" Nov 04 03:33:27.903159 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.902029 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j2z4\" (UniqueName: \"kubernetes.io/projected/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-kube-api-access-2j2z4\") pod \"e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw\" (UID: \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\") " pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" Nov 04 03:33:27.903159 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.902078 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-workload-socket\") pod \"e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw\" (UID: \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\") " pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" Nov 04 03:33:27.903159 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.902113 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-workload-certs\") pod \"e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw\" (UID: \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\") " pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" Nov 04 03:33:27.903159 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.902228 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-istiod-ca-cert\") pod \"e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw\" (UID: \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\") " pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" Nov 04 03:33:27.903159 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.902264 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-istio-token\") pod \"e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw\" (UID: \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\") " pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" Nov 04 03:33:27.903159 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.902311 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-credential-socket\") pod \"e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw\" (UID: \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\") " pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" Nov 04 03:33:27.903159 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.902372 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-istio-envoy\") pod \"e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw\" (UID: \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\") " pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" Nov 04 03:33:27.903159 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.902422 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-istio-podinfo\") pod \"e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw\" (UID: \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\") " pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" Nov 04 03:33:27.919770 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.919707 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5bd7575-de37-42e3-8b0a-75a5be69eae8-kube-api-access-rvpvn" pod "a5bd7575-de37-42e3-8b0a-75a5be69eae8" (UID: "a5bd7575-de37-42e3-8b0a-75a5be69eae8"). InnerVolumeSpecName "kube-api-access-rvpvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:33:27.924972 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.923212 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw"] Nov 04 03:33:27.941554 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.941530 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7379-d99e5f" Nov 04 03:33:27.961776 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:27.961740 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-9dda20-openshift-default-dockercfg-9kh4h\"" Nov 04 03:33:28.003216 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.003177 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4826b6b-4ae9-4947-891b-eb8dd25306cc-kube-api-access-4dv8j\" (UniqueName: \"kubernetes.io/projected/a4826b6b-4ae9-4947-891b-eb8dd25306cc-kube-api-access-4dv8j\") pod \"a4826b6b-4ae9-4947-891b-eb8dd25306cc\" (UID: \"a4826b6b-4ae9-4947-891b-eb8dd25306cc\") " Nov 04 03:33:28.003527 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.003494 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-istio-data\") pod \"e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw\" (UID: \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\") " pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" Nov 04 03:33:28.003663 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.003601 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j2z4\" (UniqueName: \"kubernetes.io/projected/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-kube-api-access-2j2z4\") pod \"e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw\" (UID: \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\") " pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" Nov 04 03:33:28.003663 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.003647 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-workload-socket\") pod \"e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw\" (UID: \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\") " pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" Nov 04 03:33:28.003817 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.003677 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-workload-certs\") pod \"e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw\" (UID: \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\") " pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" Nov 04 03:33:28.003817 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.003752 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-istiod-ca-cert\") pod \"e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw\" (UID: \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\") " pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" Nov 04 03:33:28.003817 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.003786 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-istio-token\") pod \"e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw\" (UID: \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\") " pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" Nov 04 03:33:28.004027 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.003829 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-credential-socket\") pod \"e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw\" (UID: \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\") " pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" Nov 04 03:33:28.004185 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.004119 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-istio-envoy\") pod \"e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw\" (UID: \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\") " pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" Nov 04 03:33:28.004229 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.004185 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-istio-podinfo\") pod \"e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw\" (UID: \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\") " pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" Nov 04 03:33:28.004269 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.004257 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-credential-socket\") pod \"e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw\" (UID: \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\") " pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" Nov 04 03:33:28.005652 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.004535 5953 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvpvn\" (UniqueName: \"kubernetes.io/projected/a5bd7575-de37-42e3-8b0a-75a5be69eae8-kube-api-access-rvpvn\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:33:28.005652 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.004884 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-istio-data\") pod \"e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw\" (UID: \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\") " pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" Nov 04 03:33:28.005652 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.005241 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-istiod-ca-cert\") pod \"e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw\" (UID: \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\") " pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" Nov 04 03:33:28.005652 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.005442 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-workload-certs\") pod \"e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw\" (UID: \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\") " pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" Nov 04 03:33:28.005652 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.005606 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-workload-socket\") pod \"e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw\" (UID: \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\") " pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" Nov 04 03:33:28.006980 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.006959 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4826b6b-4ae9-4947-891b-eb8dd25306cc-kube-api-access-4dv8j" pod "a4826b6b-4ae9-4947-891b-eb8dd25306cc" (UID: "a4826b6b-4ae9-4947-891b-eb8dd25306cc"). InnerVolumeSpecName "kube-api-access-4dv8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:33:28.007416 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.007383 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-istio-envoy\") pod \"e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw\" (UID: \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\") " pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" Nov 04 03:33:28.007498 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.007461 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-istio-podinfo\") pod \"e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw\" (UID: \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\") " pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" Nov 04 03:33:28.067316 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.067225 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-istio-token\") pod \"e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw\" (UID: \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\") " pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" Nov 04 03:33:28.067506 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.067337 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j2z4\" (UniqueName: \"kubernetes.io/projected/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-kube-api-access-2j2z4\") pod \"e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw\" (UID: \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\") " pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" Nov 04 03:33:28.105353 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.105320 5953 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dv8j\" (UniqueName: \"kubernetes.io/projected/a4826b6b-4ae9-4947-891b-eb8dd25306cc-kube-api-access-4dv8j\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:33:28.188747 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.188715 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" Nov 04 03:33:28.284728 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.284686 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4"] Nov 04 03:33:28.289711 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:33:28.289677 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48e61829_0b7d_40a9_b877_398aa633b44f.slice/crio-5d638e5d4dfe057c4583163546e3a4e1f71a2ce97249cf086cf3d9d1d36d423e WatchSource:0}: Error finding container 5d638e5d4dfe057c4583163546e3a4e1f71a2ce97249cf086cf3d9d1d36d423e: Status 404 returned error can't find the container with id 5d638e5d4dfe057c4583163546e3a4e1f71a2ce97249cf086cf3d9d1d36d423e Nov 04 03:33:28.292265 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.292233 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:33:28.292359 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.292298 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:33:28.292359 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.292330 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:33:28.580047 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.580013 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw"] Nov 04 03:33:28.581962 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:33:28.581934 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod587e29e9_ccc7_4343_8dd3_2fd2088f5d6c.slice/crio-ccb0f7a5ba6a6bafe42fa99c772e14646e37339318e7998362698d814fabc42e WatchSource:0}: Error finding container ccb0f7a5ba6a6bafe42fa99c772e14646e37339318e7998362698d814fabc42e: Status 404 returned error can't find the container with id ccb0f7a5ba6a6bafe42fa99c772e14646e37339318e7998362698d814fabc42e Nov 04 03:33:28.585413 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.585378 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:33:28.585495 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.585467 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:33:28.585546 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.585507 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:33:28.757505 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.757460 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" event={"ID":"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c","Type":"ContainerStarted","Data":"6d0ee94c568256be6c556c2efa13f97288654fdf616fa7ed50a50af976645edb"} Nov 04 03:33:28.757741 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.757563 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" event={"ID":"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c","Type":"ContainerStarted","Data":"ccb0f7a5ba6a6bafe42fa99c772e14646e37339318e7998362698d814fabc42e"} Nov 04 03:33:28.759765 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.759729 5953 generic.go:356] "Generic (PLEG): container finished" podID="a4826b6b-4ae9-4947-891b-eb8dd25306cc" containerID="b4b58913af50a91108edb5aacb7a21220c42c6e1834f03974240c98cf4c7a356" exitCode=137 Nov 04 03:33:28.759895 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.759776 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7379-d99e5f" Nov 04 03:33:28.759895 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.759879 5953 scope.go:122] "RemoveContainer" containerID="b4b58913af50a91108edb5aacb7a21220c42c6e1834f03974240c98cf4c7a356" Nov 04 03:33:28.766085 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.766051 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" event={"ID":"48e61829-0b7d-40a9-b877-398aa633b44f","Type":"ContainerStarted","Data":"39fdaf600df1f46a9de017426ecd1afdb4ccbbcc89b7d94bd1dc75887e08c9e8"} Nov 04 03:33:28.766215 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.766128 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" event={"ID":"48e61829-0b7d-40a9-b877-398aa633b44f","Type":"ContainerStarted","Data":"5d638e5d4dfe057c4583163546e3a4e1f71a2ce97249cf086cf3d9d1d36d423e"} Nov 04 03:33:28.768985 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.768960 5953 generic.go:356] "Generic (PLEG): container finished" podID="a5bd7575-de37-42e3-8b0a-75a5be69eae8" containerID="55c743562a8857d34eba31751c6f32c2f0ae5561c6858e2e89ba03c381a58757" exitCode=137 Nov 04 03:33:28.769114 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.769010 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7388-12987b" Nov 04 03:33:28.778813 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.778793 5953 scope.go:122] "RemoveContainer" containerID="b4b58913af50a91108edb5aacb7a21220c42c6e1834f03974240c98cf4c7a356" Nov 04 03:33:28.779121 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:33:28.779101 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4b58913af50a91108edb5aacb7a21220c42c6e1834f03974240c98cf4c7a356\": container with ID starting with b4b58913af50a91108edb5aacb7a21220c42c6e1834f03974240c98cf4c7a356 not found: ID does not exist" containerID="b4b58913af50a91108edb5aacb7a21220c42c6e1834f03974240c98cf4c7a356" Nov 04 03:33:28.779169 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.779130 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4b58913af50a91108edb5aacb7a21220c42c6e1834f03974240c98cf4c7a356"} err="failed to get container status \"b4b58913af50a91108edb5aacb7a21220c42c6e1834f03974240c98cf4c7a356\": rpc error: code = NotFound desc = could not find container \"b4b58913af50a91108edb5aacb7a21220c42c6e1834f03974240c98cf4c7a356\": container with ID starting with b4b58913af50a91108edb5aacb7a21220c42c6e1834f03974240c98cf4c7a356 not found: ID does not exist" Nov 04 03:33:28.779169 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.779148 5953 scope.go:122] "RemoveContainer" containerID="55c743562a8857d34eba31751c6f32c2f0ae5561c6858e2e89ba03c381a58757" Nov 04 03:33:28.788498 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.788477 5953 scope.go:122] "RemoveContainer" containerID="55c743562a8857d34eba31751c6f32c2f0ae5561c6858e2e89ba03c381a58757" Nov 04 03:33:28.788892 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:33:28.788870 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55c743562a8857d34eba31751c6f32c2f0ae5561c6858e2e89ba03c381a58757\": container with ID starting with 55c743562a8857d34eba31751c6f32c2f0ae5561c6858e2e89ba03c381a58757 not found: ID does not exist" containerID="55c743562a8857d34eba31751c6f32c2f0ae5561c6858e2e89ba03c381a58757" Nov 04 03:33:28.788943 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.788899 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55c743562a8857d34eba31751c6f32c2f0ae5561c6858e2e89ba03c381a58757"} err="failed to get container status \"55c743562a8857d34eba31751c6f32c2f0ae5561c6858e2e89ba03c381a58757\": rpc error: code = NotFound desc = could not find container \"55c743562a8857d34eba31751c6f32c2f0ae5561c6858e2e89ba03c381a58757\": container with ID starting with 55c743562a8857d34eba31751c6f32c2f0ae5561c6858e2e89ba03c381a58757 not found: ID does not exist" Nov 04 03:33:28.830107 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.830021 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" Nov 04 03:33:28.857167 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.857129 5953 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4826b6b-4ae9-4947-891b-eb8dd25306cc" path="/var/lib/kubelet/pods/a4826b6b-4ae9-4947-891b-eb8dd25306cc/volumes" Nov 04 03:33:28.857707 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.857687 5953 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5bd7575-de37-42e3-8b0a-75a5be69eae8" path="/var/lib/kubelet/pods/a5bd7575-de37-42e3-8b0a-75a5be69eae8/volumes" Nov 04 03:33:28.879445 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:28.879125 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" podStartSLOduration=1.879078422 podStartE2EDuration="1.879078422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:33:28.878337588 +0000 UTC m=+934.152968351" watchObservedRunningTime="2026-11-04 03:33:28.879078422 +0000 UTC m=+934.153709186" Nov 04 03:33:29.003023 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:29.000523 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" podStartSLOduration=2.000506657 podStartE2EDuration="2.000506657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:33:28.998732123 +0000 UTC m=+934.273362885" watchObservedRunningTime="2026-11-04 03:33:29.000506657 +0000 UTC m=+934.275137419" Nov 04 03:33:29.189695 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:29.189599 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" Nov 04 03:33:29.195361 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:29.195332 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" Nov 04 03:33:29.777247 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:29.777217 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" Nov 04 03:33:29.778086 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:29.778067 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" Nov 04 03:33:29.835732 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:29.835702 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" Nov 04 03:33:30.782973 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:30.782941 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" Nov 04 03:33:30.784047 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:33:30.784024 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" Nov 04 03:33:46.384470 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:33:46.384435 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:34:03.664851 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:03.664812 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp"] Nov 04 03:34:03.672429 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:03.672407 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp" Nov 04 03:34:03.699398 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:03.699368 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/264f005e-1262-4db5-915b-4887b1425904-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp\" (UID: \"264f005e-1262-4db5-915b-4887b1425904\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp" Nov 04 03:34:03.699568 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:03.699437 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/264f005e-1262-4db5-915b-4887b1425904-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp\" (UID: \"264f005e-1262-4db5-915b-4887b1425904\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp" Nov 04 03:34:03.699614 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:03.699557 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/264f005e-1262-4db5-915b-4887b1425904-metrics-certs\") pod \"maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp\" (UID: \"264f005e-1262-4db5-915b-4887b1425904\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp" Nov 04 03:34:03.699614 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:03.699600 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rrcf\" (UniqueName: \"kubernetes.io/projected/264f005e-1262-4db5-915b-4887b1425904-kube-api-access-9rrcf\") pod \"maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp\" (UID: \"264f005e-1262-4db5-915b-4887b1425904\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp" Nov 04 03:34:03.775491 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:03.775456 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-9dda20-serving-cert\"" Nov 04 03:34:03.777499 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:03.777479 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp"] Nov 04 03:34:03.801002 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:03.800964 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/264f005e-1262-4db5-915b-4887b1425904-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp\" (UID: \"264f005e-1262-4db5-915b-4887b1425904\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp" Nov 04 03:34:03.801265 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:03.801093 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/264f005e-1262-4db5-915b-4887b1425904-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp\" (UID: \"264f005e-1262-4db5-915b-4887b1425904\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp" Nov 04 03:34:03.801265 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:34:03.801236 5953 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-9dda20-serving-cert: secret "maas-api-e2e-extmod-a-9dda20-serving-cert" not found Nov 04 03:34:03.801490 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:34:03.801318 5953 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/264f005e-1262-4db5-915b-4887b1425904-maas-api-tls podName:264f005e-1262-4db5-915b-4887b1425904 nodeName:}" failed. No retries permitted until 2026-11-04 03:34:04.301292737 +0000 UTC m=+969.575923481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/264f005e-1262-4db5-915b-4887b1425904-maas-api-tls") pod "maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp" (UID: "264f005e-1262-4db5-915b-4887b1425904") : secret "maas-api-e2e-extmod-a-9dda20-serving-cert" not found Nov 04 03:34:03.801691 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:03.801667 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/264f005e-1262-4db5-915b-4887b1425904-metrics-certs\") pod \"maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp\" (UID: \"264f005e-1262-4db5-915b-4887b1425904\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp" Nov 04 03:34:03.801799 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:03.801725 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rrcf\" (UniqueName: \"kubernetes.io/projected/264f005e-1262-4db5-915b-4887b1425904-kube-api-access-9rrcf\") pod \"maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp\" (UID: \"264f005e-1262-4db5-915b-4887b1425904\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp" Nov 04 03:34:03.801864 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:03.801839 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/264f005e-1262-4db5-915b-4887b1425904-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp\" (UID: \"264f005e-1262-4db5-915b-4887b1425904\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp" Nov 04 03:34:03.804464 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:03.804437 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/264f005e-1262-4db5-915b-4887b1425904-metrics-certs\") pod \"maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp\" (UID: \"264f005e-1262-4db5-915b-4887b1425904\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp" Nov 04 03:34:03.864632 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:03.864593 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rrcf\" (UniqueName: \"kubernetes.io/projected/264f005e-1262-4db5-915b-4887b1425904-kube-api-access-9rrcf\") pod \"maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp\" (UID: \"264f005e-1262-4db5-915b-4887b1425904\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp" Nov 04 03:34:04.308154 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:04.308113 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/264f005e-1262-4db5-915b-4887b1425904-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp\" (UID: \"264f005e-1262-4db5-915b-4887b1425904\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp" Nov 04 03:34:04.310676 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:04.310644 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/264f005e-1262-4db5-915b-4887b1425904-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp\" (UID: \"264f005e-1262-4db5-915b-4887b1425904\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp" Nov 04 03:34:04.586673 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:04.586530 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp" Nov 04 03:34:04.965716 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:04.965691 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp"] Nov 04 03:34:04.968502 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:34:04.968470 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod264f005e_1262_4db5_915b_4887b1425904.slice/crio-076135260a1c40915102e30cad30288d7bce8b72cd735b1582e2e290a81b747c WatchSource:0}: Error finding container 076135260a1c40915102e30cad30288d7bce8b72cd735b1582e2e290a81b747c: Status 404 returned error can't find the container with id 076135260a1c40915102e30cad30288d7bce8b72cd735b1582e2e290a81b747c Nov 04 03:34:04.977792 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:04.977763 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp" event={"ID":"264f005e-1262-4db5-915b-4887b1425904","Type":"ContainerStarted","Data":"076135260a1c40915102e30cad30288d7bce8b72cd735b1582e2e290a81b747c"} Nov 04 03:34:05.984459 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:05.984421 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp" event={"ID":"264f005e-1262-4db5-915b-4887b1425904","Type":"ContainerStarted","Data":"52bbc76a7c3a641e3281565a9bf60af625c6a31b90b2aa25850f0cbd0bcb0859"} Nov 04 03:34:05.984894 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:05.984644 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp" Nov 04 03:34:06.111631 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:06.111577 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp" podStartSLOduration=2.206963995 podStartE2EDuration="3.111560761s" totalImagesPullingTime="904.596766ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:34:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:34:06.109973033 +0000 UTC m=+971.384603809" watchObservedRunningTime="2026-11-04 03:34:06.111560761 +0000 UTC m=+971.386191522" Nov 04 03:34:08.589765 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:08.589727 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976"] Nov 04 03:34:08.593748 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:08.593726 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976" Nov 04 03:34:08.646342 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:08.646305 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-141891-serving-cert\"" Nov 04 03:34:08.658520 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:08.658481 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3220ff1-68e3-4568-a53a-4a9f6e06a5ad-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976\" (UID: \"d3220ff1-68e3-4568-a53a-4a9f6e06a5ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976" Nov 04 03:34:08.658716 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:08.658548 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3220ff1-68e3-4568-a53a-4a9f6e06a5ad-maas-api-tls\") pod \"maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976\" (UID: \"d3220ff1-68e3-4568-a53a-4a9f6e06a5ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976" Nov 04 03:34:08.658803 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:08.658772 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3220ff1-68e3-4568-a53a-4a9f6e06a5ad-metrics-certs\") pod \"maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976\" (UID: \"d3220ff1-68e3-4568-a53a-4a9f6e06a5ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976" Nov 04 03:34:08.658923 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:08.658883 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjvmq\" (UniqueName: \"kubernetes.io/projected/d3220ff1-68e3-4568-a53a-4a9f6e06a5ad-kube-api-access-tjvmq\") pod \"maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976\" (UID: \"d3220ff1-68e3-4568-a53a-4a9f6e06a5ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976" Nov 04 03:34:08.698754 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:08.698719 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976"] Nov 04 03:34:08.759956 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:08.759918 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3220ff1-68e3-4568-a53a-4a9f6e06a5ad-metrics-certs\") pod \"maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976\" (UID: \"d3220ff1-68e3-4568-a53a-4a9f6e06a5ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976" Nov 04 03:34:08.760135 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:08.759972 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjvmq\" (UniqueName: \"kubernetes.io/projected/d3220ff1-68e3-4568-a53a-4a9f6e06a5ad-kube-api-access-tjvmq\") pod \"maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976\" (UID: \"d3220ff1-68e3-4568-a53a-4a9f6e06a5ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976" Nov 04 03:34:08.760135 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:08.760020 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3220ff1-68e3-4568-a53a-4a9f6e06a5ad-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976\" (UID: \"d3220ff1-68e3-4568-a53a-4a9f6e06a5ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976" Nov 04 03:34:08.760135 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:08.760041 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3220ff1-68e3-4568-a53a-4a9f6e06a5ad-maas-api-tls\") pod \"maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976\" (UID: \"d3220ff1-68e3-4568-a53a-4a9f6e06a5ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976" Nov 04 03:34:08.760254 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:34:08.760220 5953 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-141891-serving-cert: secret "maas-api-e2e-extmod-a-141891-serving-cert" not found Nov 04 03:34:08.760333 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:34:08.760319 5953 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3220ff1-68e3-4568-a53a-4a9f6e06a5ad-maas-api-tls podName:d3220ff1-68e3-4568-a53a-4a9f6e06a5ad nodeName:}" failed. No retries permitted until 2026-11-04 03:34:09.26028224 +0000 UTC m=+974.534912980 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d3220ff1-68e3-4568-a53a-4a9f6e06a5ad-maas-api-tls") pod "maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976" (UID: "d3220ff1-68e3-4568-a53a-4a9f6e06a5ad") : secret "maas-api-e2e-extmod-a-141891-serving-cert" not found Nov 04 03:34:08.760673 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:08.760649 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3220ff1-68e3-4568-a53a-4a9f6e06a5ad-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976\" (UID: \"d3220ff1-68e3-4568-a53a-4a9f6e06a5ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976" Nov 04 03:34:08.762578 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:08.762561 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3220ff1-68e3-4568-a53a-4a9f6e06a5ad-metrics-certs\") pod \"maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976\" (UID: \"d3220ff1-68e3-4568-a53a-4a9f6e06a5ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976" Nov 04 03:34:08.823861 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:08.823830 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjvmq\" (UniqueName: \"kubernetes.io/projected/d3220ff1-68e3-4568-a53a-4a9f6e06a5ad-kube-api-access-tjvmq\") pod \"maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976\" (UID: \"d3220ff1-68e3-4568-a53a-4a9f6e06a5ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976" Nov 04 03:34:09.265165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:09.265131 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3220ff1-68e3-4568-a53a-4a9f6e06a5ad-maas-api-tls\") pod \"maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976\" (UID: \"d3220ff1-68e3-4568-a53a-4a9f6e06a5ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976" Nov 04 03:34:09.267720 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:09.267691 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3220ff1-68e3-4568-a53a-4a9f6e06a5ad-maas-api-tls\") pod \"maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976\" (UID: \"d3220ff1-68e3-4568-a53a-4a9f6e06a5ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976" Nov 04 03:34:09.507815 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:09.507774 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976" Nov 04 03:34:09.886998 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:09.886969 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976"] Nov 04 03:34:09.889979 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:34:09.889925 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3220ff1_68e3_4568_a53a_4a9f6e06a5ad.slice/crio-b3f2de689d98c0bf2b0994267896c98649f0b0c1dc8ae09bc4cb68c3967a8bb3 WatchSource:0}: Error finding container b3f2de689d98c0bf2b0994267896c98649f0b0c1dc8ae09bc4cb68c3967a8bb3: Status 404 returned error can't find the container with id b3f2de689d98c0bf2b0994267896c98649f0b0c1dc8ae09bc4cb68c3967a8bb3 Nov 04 03:34:10.007501 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:10.007467 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976" event={"ID":"d3220ff1-68e3-4568-a53a-4a9f6e06a5ad","Type":"ContainerStarted","Data":"b3f2de689d98c0bf2b0994267896c98649f0b0c1dc8ae09bc4cb68c3967a8bb3"} Nov 04 03:34:11.014868 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:11.014776 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976" event={"ID":"d3220ff1-68e3-4568-a53a-4a9f6e06a5ad","Type":"ContainerStarted","Data":"bd854bed2ecb4436e10137b49a38ac8649af6b5ba7fc1acf8860bc53176abdbf"} Nov 04 03:34:11.014868 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:11.014853 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976" Nov 04 03:34:11.134882 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:11.134816 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976" podStartSLOduration=2.265821017 podStartE2EDuration="3.134796258s" totalImagesPullingTime="868.975241ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:34:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:34:11.132856569 +0000 UTC m=+976.407487332" watchObservedRunningTime="2026-11-04 03:34:11.134796258 +0000 UTC m=+976.409427021" Nov 04 03:34:11.995365 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:11.995337 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp" Nov 04 03:34:16.385401 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:34:16.385354 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:34:17.031682 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:17.031655 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976" Nov 04 03:34:23.087707 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:23.087670 5953 generic.go:356] "Generic (PLEG): container finished" podID="587e29e9-ccc7-4343-8dd3-2fd2088f5d6c" containerID="6d0ee94c568256be6c556c2efa13f97288654fdf616fa7ed50a50af976645edb" exitCode=0 Nov 04 03:34:23.088246 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:23.087750 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" event={"ID":"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c","Type":"ContainerDied","Data":"6d0ee94c568256be6c556c2efa13f97288654fdf616fa7ed50a50af976645edb"} Nov 04 03:34:23.088246 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:23.088241 5953 scope.go:122] "RemoveContainer" containerID="6d0ee94c568256be6c556c2efa13f97288654fdf616fa7ed50a50af976645edb" Nov 04 03:34:23.089808 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:23.089779 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:34:23.089928 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:23.089862 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:34:23.089928 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:23.089909 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:34:24.094899 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:24.094868 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" event={"ID":"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c","Type":"ContainerStarted","Data":"c4d050c2e7885c80a350a02927b7f93ac2b98e4612d814c71883ee8b56ccbc81"} Nov 04 03:34:24.189173 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:24.189135 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" Nov 04 03:34:24.189357 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:24.189297 5953 patch_prober.go:29] interesting pod/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 04 03:34:24.189403 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:24.189350 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" podUID="587e29e9-ccc7-4343-8dd3-2fd2088f5d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 04 03:34:25.189135 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:25.189098 5953 patch_prober.go:29] interesting pod/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 04 03:34:25.189660 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:25.189169 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" podUID="587e29e9-ccc7-4343-8dd3-2fd2088f5d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 04 03:34:26.189489 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:26.189457 5953 patch_prober.go:29] interesting pod/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 04 03:34:26.189965 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:26.189524 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" podUID="587e29e9-ccc7-4343-8dd3-2fd2088f5d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 04 03:34:27.189762 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:27.189731 5953 patch_prober.go:29] interesting pod/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 04 03:34:27.190247 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:27.189793 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" podUID="587e29e9-ccc7-4343-8dd3-2fd2088f5d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 04 03:34:28.189644 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:28.189588 5953 patch_prober.go:29] interesting pod/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 04 03:34:28.189841 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:28.189684 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" podUID="587e29e9-ccc7-4343-8dd3-2fd2088f5d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 04 03:34:29.189056 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:29.189019 5953 patch_prober.go:29] interesting pod/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 04 03:34:29.189246 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:29.189075 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" podUID="587e29e9-ccc7-4343-8dd3-2fd2088f5d6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 04 03:34:29.777127 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:29.777095 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" Nov 04 03:34:30.193934 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:30.193851 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" Nov 04 03:34:30.194969 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:30.194950 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" Nov 04 03:34:41.819443 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:41.819393 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-36ccd3/distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd"] Nov 04 03:34:41.829059 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:41.829033 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-36ccd3/distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd" Nov 04 03:34:41.882229 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:41.882191 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-36ccd3/distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd"] Nov 04 03:34:41.899804 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:41.899766 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg44c\" (UniqueName: \"kubernetes.io/projected/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-kube-api-access-lg44c\") pod \"distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd\" (UID: \"d5ba197d-472f-4f3c-bdd1-0b45ae01090f\") " pod="e2e-models-e2e-worker-w0-36ccd3/distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd" Nov 04 03:34:41.899804 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:41.899813 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-model-cache\") pod \"distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd\" (UID: \"d5ba197d-472f-4f3c-bdd1-0b45ae01090f\") " pod="e2e-models-e2e-worker-w0-36ccd3/distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd" Nov 04 03:34:41.900042 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:41.899898 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-tmp-dir\") pod \"distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd\" (UID: \"d5ba197d-472f-4f3c-bdd1-0b45ae01090f\") " pod="e2e-models-e2e-worker-w0-36ccd3/distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd" Nov 04 03:34:41.900042 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:41.900012 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-tls-certs\") pod \"distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd\" (UID: \"d5ba197d-472f-4f3c-bdd1-0b45ae01090f\") " pod="e2e-models-e2e-worker-w0-36ccd3/distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd" Nov 04 03:34:41.900133 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:41.900042 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-dshm\") pod \"distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd\" (UID: \"d5ba197d-472f-4f3c-bdd1-0b45ae01090f\") " pod="e2e-models-e2e-worker-w0-36ccd3/distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd" Nov 04 03:34:41.900133 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:41.900074 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-home\") pod \"distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd\" (UID: \"d5ba197d-472f-4f3c-bdd1-0b45ae01090f\") " pod="e2e-models-e2e-worker-w0-36ccd3/distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd" Nov 04 03:34:41.929806 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:41.929769 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-36ccd3\"/\"distinct-2-w0-36ccd3-kserve-self-signed-certs\"" Nov 04 03:34:42.001818 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:42.001782 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-tmp-dir\") pod \"distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd\" (UID: \"d5ba197d-472f-4f3c-bdd1-0b45ae01090f\") " pod="e2e-models-e2e-worker-w0-36ccd3/distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd" Nov 04 03:34:42.002013 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:42.001870 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-tls-certs\") pod \"distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd\" (UID: \"d5ba197d-472f-4f3c-bdd1-0b45ae01090f\") " pod="e2e-models-e2e-worker-w0-36ccd3/distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd" Nov 04 03:34:42.002013 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:42.001896 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-dshm\") pod \"distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd\" (UID: \"d5ba197d-472f-4f3c-bdd1-0b45ae01090f\") " pod="e2e-models-e2e-worker-w0-36ccd3/distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd" Nov 04 03:34:42.002013 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:42.001938 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-home\") pod \"distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd\" (UID: \"d5ba197d-472f-4f3c-bdd1-0b45ae01090f\") " pod="e2e-models-e2e-worker-w0-36ccd3/distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd" Nov 04 03:34:42.002189 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:42.002092 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg44c\" (UniqueName: \"kubernetes.io/projected/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-kube-api-access-lg44c\") pod \"distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd\" (UID: \"d5ba197d-472f-4f3c-bdd1-0b45ae01090f\") " pod="e2e-models-e2e-worker-w0-36ccd3/distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd" Nov 04 03:34:42.002189 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:42.002123 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-model-cache\") pod \"distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd\" (UID: \"d5ba197d-472f-4f3c-bdd1-0b45ae01090f\") " pod="e2e-models-e2e-worker-w0-36ccd3/distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd" Nov 04 03:34:42.002296 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:42.002227 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-tmp-dir\") pod \"distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd\" (UID: \"d5ba197d-472f-4f3c-bdd1-0b45ae01090f\") " pod="e2e-models-e2e-worker-w0-36ccd3/distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd" Nov 04 03:34:42.002296 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:42.002281 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-home\") pod \"distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd\" (UID: \"d5ba197d-472f-4f3c-bdd1-0b45ae01090f\") " pod="e2e-models-e2e-worker-w0-36ccd3/distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd" Nov 04 03:34:42.002475 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:42.002456 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-model-cache\") pod \"distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd\" (UID: \"d5ba197d-472f-4f3c-bdd1-0b45ae01090f\") " pod="e2e-models-e2e-worker-w0-36ccd3/distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd" Nov 04 03:34:42.004283 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:42.004265 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-dshm\") pod \"distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd\" (UID: \"d5ba197d-472f-4f3c-bdd1-0b45ae01090f\") " pod="e2e-models-e2e-worker-w0-36ccd3/distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd" Nov 04 03:34:42.004562 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:42.004545 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-tls-certs\") pod \"distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd\" (UID: \"d5ba197d-472f-4f3c-bdd1-0b45ae01090f\") " pod="e2e-models-e2e-worker-w0-36ccd3/distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd" Nov 04 03:34:42.063235 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:42.063205 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg44c\" (UniqueName: \"kubernetes.io/projected/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-kube-api-access-lg44c\") pod \"distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd\" (UID: \"d5ba197d-472f-4f3c-bdd1-0b45ae01090f\") " pod="e2e-models-e2e-worker-w0-36ccd3/distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd" Nov 04 03:34:42.141723 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:42.141579 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-36ccd3/distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd" Nov 04 03:34:42.243821 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:42.243788 5953 generic.go:356] "Generic (PLEG): container finished" podID="48e61829-0b7d-40a9-b877-398aa633b44f" containerID="39fdaf600df1f46a9de017426ecd1afdb4ccbbcc89b7d94bd1dc75887e08c9e8" exitCode=0 Nov 04 03:34:42.243975 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:42.243868 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" event={"ID":"48e61829-0b7d-40a9-b877-398aa633b44f","Type":"ContainerDied","Data":"39fdaf600df1f46a9de017426ecd1afdb4ccbbcc89b7d94bd1dc75887e08c9e8"} Nov 04 03:34:42.244238 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:42.244224 5953 scope.go:122] "RemoveContainer" containerID="39fdaf600df1f46a9de017426ecd1afdb4ccbbcc89b7d94bd1dc75887e08c9e8" Nov 04 03:34:42.245731 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:42.245706 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:34:42.245800 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:42.245787 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:34:42.245868 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:42.245818 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:34:42.535845 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:42.535716 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-36ccd3/distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd"] Nov 04 03:34:42.538255 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:34:42.538226 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5ba197d_472f_4f3c_bdd1_0b45ae01090f.slice/crio-9fb62641e54b4ef0546e59f68d4ce503c5017984337d9e418adc12087af0884c WatchSource:0}: Error finding container 9fb62641e54b4ef0546e59f68d4ce503c5017984337d9e418adc12087af0884c: Status 404 returned error can't find the container with id 9fb62641e54b4ef0546e59f68d4ce503c5017984337d9e418adc12087af0884c Nov 04 03:34:43.252735 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:43.252691 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" event={"ID":"48e61829-0b7d-40a9-b877-398aa633b44f","Type":"ContainerStarted","Data":"f26695f45bd5653f31814c306f3ca1242d8e3f2ed21ff63550d59b0dbc33dc0d"} Nov 04 03:34:43.254907 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:43.254878 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-36ccd3/distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd" event={"ID":"d5ba197d-472f-4f3c-bdd1-0b45ae01090f","Type":"ContainerStarted","Data":"d46b6b4e84ae4cd9d98d8f2ba89421d423935315e7cb65c595e695b33390fd73"} Nov 04 03:34:43.254907 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:43.254919 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-36ccd3/distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd" event={"ID":"d5ba197d-472f-4f3c-bdd1-0b45ae01090f","Type":"ContainerStarted","Data":"9fb62641e54b4ef0546e59f68d4ce503c5017984337d9e418adc12087af0884c"} Nov 04 03:34:43.495150 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:43.495093 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-36ccd3/distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd" podStartSLOduration=2.495078051 podStartE2EDuration="2.495078051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:34:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:34:43.494231082 +0000 UTC m=+1008.768861843" watchObservedRunningTime="2026-11-04 03:34:43.495078051 +0000 UTC m=+1008.769708807" Nov 04 03:34:43.833784 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:43.830722 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" Nov 04 03:34:43.833784 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:43.831092 5953 patch_prober.go:29] interesting pod/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 04 03:34:43.833784 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:43.831174 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" podUID="48e61829-0b7d-40a9-b877-398aa633b44f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 04 03:34:44.142002 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:44.141919 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-36ccd3/distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd" Nov 04 03:34:44.833649 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:44.832044 5953 patch_prober.go:29] interesting pod/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 04 03:34:44.833649 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:44.832134 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" podUID="48e61829-0b7d-40a9-b877-398aa633b44f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 04 03:34:45.782751 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:45.782707 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" Nov 04 03:34:45.834646 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:45.831138 5953 patch_prober.go:29] interesting pod/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 04 03:34:45.834646 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:45.831225 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" podUID="48e61829-0b7d-40a9-b877-398aa633b44f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 04 03:34:46.389712 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:34:46.389526 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:34:46.835820 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:46.833098 5953 patch_prober.go:29] interesting pod/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 04 03:34:46.835820 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:46.833158 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" podUID="48e61829-0b7d-40a9-b877-398aa633b44f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 04 03:34:47.830019 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:47.829984 5953 patch_prober.go:29] interesting pod/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 04 03:34:47.830378 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:47.830353 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" podUID="48e61829-0b7d-40a9-b877-398aa633b44f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 04 03:34:48.895048 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:48.895005 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" Nov 04 03:34:48.896732 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:48.896696 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" Nov 04 03:34:52.146032 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:52.146004 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-36ccd3/distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd" Nov 04 03:34:52.200534 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:52.200497 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-36ccd3/distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd" Nov 04 03:34:52.395455 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:34:52.395423 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-36ccd3/distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd" Nov 04 03:35:16.385688 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:35:16.385613 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:35:23.801646 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:23.801601 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6"] Nov 04 03:35:23.804721 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:23.804706 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" Nov 04 03:35:23.857685 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:23.857605 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-36ccd3-openshift-default-dockercfg-n5hpn\"" Nov 04 03:35:23.932165 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:23.932122 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6"] Nov 04 03:35:23.987955 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:23.987914 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58497b8e-768f-4133-8c04-a001d4af6ca5-istio-token\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6\" (UID: \"58497b8e-768f-4133-8c04-a001d4af6ca5\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" Nov 04 03:35:23.988136 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:23.987967 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58497b8e-768f-4133-8c04-a001d4af6ca5-credential-socket\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6\" (UID: \"58497b8e-768f-4133-8c04-a001d4af6ca5\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" Nov 04 03:35:23.988136 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:23.987992 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58497b8e-768f-4133-8c04-a001d4af6ca5-workload-socket\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6\" (UID: \"58497b8e-768f-4133-8c04-a001d4af6ca5\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" Nov 04 03:35:23.988217 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:23.988198 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58497b8e-768f-4133-8c04-a001d4af6ca5-istio-data\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6\" (UID: \"58497b8e-768f-4133-8c04-a001d4af6ca5\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" Nov 04 03:35:23.988275 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:23.988255 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58497b8e-768f-4133-8c04-a001d4af6ca5-istio-envoy\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6\" (UID: \"58497b8e-768f-4133-8c04-a001d4af6ca5\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" Nov 04 03:35:23.988332 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:23.988315 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58497b8e-768f-4133-8c04-a001d4af6ca5-istiod-ca-cert\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6\" (UID: \"58497b8e-768f-4133-8c04-a001d4af6ca5\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" Nov 04 03:35:23.988409 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:23.988384 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58497b8e-768f-4133-8c04-a001d4af6ca5-workload-certs\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6\" (UID: \"58497b8e-768f-4133-8c04-a001d4af6ca5\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" Nov 04 03:35:23.988455 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:23.988424 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58497b8e-768f-4133-8c04-a001d4af6ca5-istio-podinfo\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6\" (UID: \"58497b8e-768f-4133-8c04-a001d4af6ca5\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" Nov 04 03:35:23.988455 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:23.988447 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ktwn\" (UniqueName: \"kubernetes.io/projected/58497b8e-768f-4133-8c04-a001d4af6ca5-kube-api-access-4ktwn\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6\" (UID: \"58497b8e-768f-4133-8c04-a001d4af6ca5\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" Nov 04 03:35:24.089307 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:24.089209 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58497b8e-768f-4133-8c04-a001d4af6ca5-istio-data\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6\" (UID: \"58497b8e-768f-4133-8c04-a001d4af6ca5\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" Nov 04 03:35:24.089307 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:24.089268 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58497b8e-768f-4133-8c04-a001d4af6ca5-istio-envoy\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6\" (UID: \"58497b8e-768f-4133-8c04-a001d4af6ca5\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" Nov 04 03:35:24.089307 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:24.089304 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58497b8e-768f-4133-8c04-a001d4af6ca5-istiod-ca-cert\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6\" (UID: \"58497b8e-768f-4133-8c04-a001d4af6ca5\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" Nov 04 03:35:24.089668 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:24.089320 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58497b8e-768f-4133-8c04-a001d4af6ca5-workload-certs\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6\" (UID: \"58497b8e-768f-4133-8c04-a001d4af6ca5\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" Nov 04 03:35:24.089668 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:24.089337 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58497b8e-768f-4133-8c04-a001d4af6ca5-istio-podinfo\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6\" (UID: \"58497b8e-768f-4133-8c04-a001d4af6ca5\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" Nov 04 03:35:24.089668 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:24.089474 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ktwn\" (UniqueName: \"kubernetes.io/projected/58497b8e-768f-4133-8c04-a001d4af6ca5-kube-api-access-4ktwn\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6\" (UID: \"58497b8e-768f-4133-8c04-a001d4af6ca5\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" Nov 04 03:35:24.089668 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:24.089589 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58497b8e-768f-4133-8c04-a001d4af6ca5-istio-token\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6\" (UID: \"58497b8e-768f-4133-8c04-a001d4af6ca5\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" Nov 04 03:35:24.090122 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:24.089686 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58497b8e-768f-4133-8c04-a001d4af6ca5-credential-socket\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6\" (UID: \"58497b8e-768f-4133-8c04-a001d4af6ca5\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" Nov 04 03:35:24.090122 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:24.089724 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58497b8e-768f-4133-8c04-a001d4af6ca5-workload-socket\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6\" (UID: \"58497b8e-768f-4133-8c04-a001d4af6ca5\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" Nov 04 03:35:24.090122 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:24.089773 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58497b8e-768f-4133-8c04-a001d4af6ca5-istio-data\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6\" (UID: \"58497b8e-768f-4133-8c04-a001d4af6ca5\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" Nov 04 03:35:24.090122 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:24.089885 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58497b8e-768f-4133-8c04-a001d4af6ca5-workload-certs\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6\" (UID: \"58497b8e-768f-4133-8c04-a001d4af6ca5\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" Nov 04 03:35:24.090122 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:24.090077 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58497b8e-768f-4133-8c04-a001d4af6ca5-workload-socket\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6\" (UID: \"58497b8e-768f-4133-8c04-a001d4af6ca5\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" Nov 04 03:35:24.090400 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:24.090191 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58497b8e-768f-4133-8c04-a001d4af6ca5-credential-socket\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6\" (UID: \"58497b8e-768f-4133-8c04-a001d4af6ca5\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" Nov 04 03:35:24.090454 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:24.090403 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58497b8e-768f-4133-8c04-a001d4af6ca5-istiod-ca-cert\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6\" (UID: \"58497b8e-768f-4133-8c04-a001d4af6ca5\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" Nov 04 03:35:24.091841 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:24.091816 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58497b8e-768f-4133-8c04-a001d4af6ca5-istio-envoy\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6\" (UID: \"58497b8e-768f-4133-8c04-a001d4af6ca5\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" Nov 04 03:35:24.092105 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:24.092086 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58497b8e-768f-4133-8c04-a001d4af6ca5-istio-podinfo\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6\" (UID: \"58497b8e-768f-4133-8c04-a001d4af6ca5\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" Nov 04 03:35:24.151700 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:24.151667 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58497b8e-768f-4133-8c04-a001d4af6ca5-istio-token\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6\" (UID: \"58497b8e-768f-4133-8c04-a001d4af6ca5\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" Nov 04 03:35:24.151892 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:24.151869 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ktwn\" (UniqueName: \"kubernetes.io/projected/58497b8e-768f-4133-8c04-a001d4af6ca5-kube-api-access-4ktwn\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6\" (UID: \"58497b8e-768f-4133-8c04-a001d4af6ca5\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" Nov 04 03:35:24.417356 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:24.417267 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" Nov 04 03:35:24.848207 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:24.848184 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6"] Nov 04 03:35:24.850531 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:35:24.850504 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58497b8e_768f_4133_8c04_a001d4af6ca5.slice/crio-f8b882dc4a187be5b623ef524efaf657a04d778f06cd42b4bcb99c9bb21f7cbb WatchSource:0}: Error finding container f8b882dc4a187be5b623ef524efaf657a04d778f06cd42b4bcb99c9bb21f7cbb: Status 404 returned error can't find the container with id f8b882dc4a187be5b623ef524efaf657a04d778f06cd42b4bcb99c9bb21f7cbb Nov 04 03:35:24.853064 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:24.853036 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:35:24.853135 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:24.853099 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:35:24.853177 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:24.853135 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:35:25.574074 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:25.574041 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" event={"ID":"58497b8e-768f-4133-8c04-a001d4af6ca5","Type":"ContainerStarted","Data":"a61332668e6b13be0a1b1de3862cf25521822ddaf6b8ac16fd5d096c5d102a43"} Nov 04 03:35:25.574264 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:25.574093 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" event={"ID":"58497b8e-768f-4133-8c04-a001d4af6ca5","Type":"ContainerStarted","Data":"f8b882dc4a187be5b623ef524efaf657a04d778f06cd42b4bcb99c9bb21f7cbb"} Nov 04 03:35:26.422500 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:26.422464 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" Nov 04 03:35:26.423245 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:26.423222 5953 patch_prober.go:29] interesting pod/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 04 03:35:26.423436 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:26.423411 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" podUID="58497b8e-768f-4133-8c04-a001d4af6ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 04 03:35:27.424650 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:27.424075 5953 patch_prober.go:29] interesting pod/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 04 03:35:27.424650 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:27.424149 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" podUID="58497b8e-768f-4133-8c04-a001d4af6ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 04 03:35:28.421652 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:28.421074 5953 patch_prober.go:29] interesting pod/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 04 03:35:28.421652 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:28.421146 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" podUID="58497b8e-768f-4133-8c04-a001d4af6ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 04 03:35:29.424057 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:29.424015 5953 patch_prober.go:29] interesting pod/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 04 03:35:29.424697 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:29.424662 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" podUID="58497b8e-768f-4133-8c04-a001d4af6ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 04 03:35:30.427825 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:30.427781 5953 patch_prober.go:29] interesting pod/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 04 03:35:30.428289 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:30.427855 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" podUID="58497b8e-768f-4133-8c04-a001d4af6ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 04 03:35:31.418382 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:31.418345 5953 patch_prober.go:29] interesting pod/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 04 03:35:31.418544 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:31.418402 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" podUID="58497b8e-768f-4133-8c04-a001d4af6ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 04 03:35:32.798035 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:32.797992 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" Nov 04 03:35:32.798879 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:32.798844 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" Nov 04 03:35:32.799883 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:32.799862 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" Nov 04 03:35:32.969690 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:32.969613 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" podStartSLOduration=9.969599976 podStartE2EDuration="9.969599976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:35:25.793952894 +0000 UTC m=+1051.068583656" watchObservedRunningTime="2026-11-04 03:35:32.969599976 +0000 UTC m=+1058.244230738" Nov 04 03:35:38.564818 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:38.564778 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g"] Nov 04 03:35:38.567598 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:38.567582 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g" Nov 04 03:35:38.620803 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:38.620768 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-13190c-serving-cert\"" Nov 04 03:35:38.671653 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:38.671596 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db0efa5b-3ced-44e4-ba5c-f3bdad377597-maas-api-tls\") pod \"maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g\" (UID: \"db0efa5b-3ced-44e4-ba5c-f3bdad377597\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g" Nov 04 03:35:38.671825 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:38.671741 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db0efa5b-3ced-44e4-ba5c-f3bdad377597-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g\" (UID: \"db0efa5b-3ced-44e4-ba5c-f3bdad377597\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g" Nov 04 03:35:38.671896 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:38.671862 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmx56\" (UniqueName: \"kubernetes.io/projected/db0efa5b-3ced-44e4-ba5c-f3bdad377597-kube-api-access-qmx56\") pod \"maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g\" (UID: \"db0efa5b-3ced-44e4-ba5c-f3bdad377597\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g" Nov 04 03:35:38.671944 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:38.671896 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db0efa5b-3ced-44e4-ba5c-f3bdad377597-metrics-certs\") pod \"maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g\" (UID: \"db0efa5b-3ced-44e4-ba5c-f3bdad377597\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g" Nov 04 03:35:38.673415 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:38.673394 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g"] Nov 04 03:35:38.773010 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:38.772969 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmx56\" (UniqueName: \"kubernetes.io/projected/db0efa5b-3ced-44e4-ba5c-f3bdad377597-kube-api-access-qmx56\") pod \"maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g\" (UID: \"db0efa5b-3ced-44e4-ba5c-f3bdad377597\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g" Nov 04 03:35:38.773180 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:38.773021 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db0efa5b-3ced-44e4-ba5c-f3bdad377597-metrics-certs\") pod \"maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g\" (UID: \"db0efa5b-3ced-44e4-ba5c-f3bdad377597\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g" Nov 04 03:35:38.773180 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:38.773082 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db0efa5b-3ced-44e4-ba5c-f3bdad377597-maas-api-tls\") pod \"maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g\" (UID: \"db0efa5b-3ced-44e4-ba5c-f3bdad377597\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g" Nov 04 03:35:38.773180 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:38.773162 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db0efa5b-3ced-44e4-ba5c-f3bdad377597-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g\" (UID: \"db0efa5b-3ced-44e4-ba5c-f3bdad377597\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g" Nov 04 03:35:38.773367 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:35:38.773248 5953 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-13190c-serving-cert: secret "maas-api-e2e-extmod-a-13190c-serving-cert" not found Nov 04 03:35:38.773367 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:35:38.773327 5953 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db0efa5b-3ced-44e4-ba5c-f3bdad377597-maas-api-tls podName:db0efa5b-3ced-44e4-ba5c-f3bdad377597 nodeName:}" failed. No retries permitted until 2026-11-04 03:35:39.273304038 +0000 UTC m=+1064.547934792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/db0efa5b-3ced-44e4-ba5c-f3bdad377597-maas-api-tls") pod "maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g" (UID: "db0efa5b-3ced-44e4-ba5c-f3bdad377597") : secret "maas-api-e2e-extmod-a-13190c-serving-cert" not found Nov 04 03:35:38.773784 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:38.773764 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db0efa5b-3ced-44e4-ba5c-f3bdad377597-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g\" (UID: \"db0efa5b-3ced-44e4-ba5c-f3bdad377597\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g" Nov 04 03:35:38.775562 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:38.775543 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db0efa5b-3ced-44e4-ba5c-f3bdad377597-metrics-certs\") pod \"maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g\" (UID: \"db0efa5b-3ced-44e4-ba5c-f3bdad377597\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g" Nov 04 03:35:38.840776 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:38.840699 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmx56\" (UniqueName: \"kubernetes.io/projected/db0efa5b-3ced-44e4-ba5c-f3bdad377597-kube-api-access-qmx56\") pod \"maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g\" (UID: \"db0efa5b-3ced-44e4-ba5c-f3bdad377597\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g" Nov 04 03:35:39.279509 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:39.279477 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db0efa5b-3ced-44e4-ba5c-f3bdad377597-maas-api-tls\") pod \"maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g\" (UID: \"db0efa5b-3ced-44e4-ba5c-f3bdad377597\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g" Nov 04 03:35:39.282024 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:39.282004 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db0efa5b-3ced-44e4-ba5c-f3bdad377597-maas-api-tls\") pod \"maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g\" (UID: \"db0efa5b-3ced-44e4-ba5c-f3bdad377597\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g" Nov 04 03:35:39.480588 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:39.480551 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g" Nov 04 03:35:39.860413 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:39.860379 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g"] Nov 04 03:35:39.863208 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:35:39.863184 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb0efa5b_3ced_44e4_ba5c_f3bdad377597.slice/crio-2fe3902d1300460854f74bd49272e5179e0efce12e5c01dd24a52b66b1702362 WatchSource:0}: Error finding container 2fe3902d1300460854f74bd49272e5179e0efce12e5c01dd24a52b66b1702362: Status 404 returned error can't find the container with id 2fe3902d1300460854f74bd49272e5179e0efce12e5c01dd24a52b66b1702362 Nov 04 03:35:40.664503 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:40.664473 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g" event={"ID":"db0efa5b-3ced-44e4-ba5c-f3bdad377597","Type":"ContainerStarted","Data":"2fe3902d1300460854f74bd49272e5179e0efce12e5c01dd24a52b66b1702362"} Nov 04 03:35:41.701350 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:41.701307 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g" event={"ID":"db0efa5b-3ced-44e4-ba5c-f3bdad377597","Type":"ContainerStarted","Data":"c6b09e9da904eaa5e56cf427947508aa0b4f03104393f458ef259e63af30ef36"} Nov 04 03:35:41.702267 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:41.702239 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g" Nov 04 03:35:43.464128 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:43.464073 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g" podStartSLOduration=4.518614137 podStartE2EDuration="5.46405583s" totalImagesPullingTime="945.441693ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:35:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:35:41.822448646 +0000 UTC m=+1067.097079409" watchObservedRunningTime="2026-11-04 03:35:43.46405583 +0000 UTC m=+1068.738686593" Nov 04 03:35:43.465386 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:43.465362 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b"] Nov 04 03:35:43.469650 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:43.469632 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b" Nov 04 03:35:43.525187 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:43.525162 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-d1e2ab-serving-cert\"" Nov 04 03:35:43.536030 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:43.536003 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6lv6\" (UniqueName: \"kubernetes.io/projected/f014fed4-d5bf-4ce6-8223-90d1d255334d-kube-api-access-c6lv6\") pod \"maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b\" (UID: \"f014fed4-d5bf-4ce6-8223-90d1d255334d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b" Nov 04 03:35:43.536183 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:43.536167 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f014fed4-d5bf-4ce6-8223-90d1d255334d-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b\" (UID: \"f014fed4-d5bf-4ce6-8223-90d1d255334d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b" Nov 04 03:35:43.536240 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:43.536220 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f014fed4-d5bf-4ce6-8223-90d1d255334d-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b\" (UID: \"f014fed4-d5bf-4ce6-8223-90d1d255334d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b" Nov 04 03:35:43.536295 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:43.536243 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f014fed4-d5bf-4ce6-8223-90d1d255334d-metrics-certs\") pod \"maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b\" (UID: \"f014fed4-d5bf-4ce6-8223-90d1d255334d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b" Nov 04 03:35:43.575871 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:43.575838 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b"] Nov 04 03:35:43.637764 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:43.637731 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f014fed4-d5bf-4ce6-8223-90d1d255334d-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b\" (UID: \"f014fed4-d5bf-4ce6-8223-90d1d255334d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b" Nov 04 03:35:43.637928 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:43.637798 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f014fed4-d5bf-4ce6-8223-90d1d255334d-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b\" (UID: \"f014fed4-d5bf-4ce6-8223-90d1d255334d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b" Nov 04 03:35:43.637928 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:43.637818 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f014fed4-d5bf-4ce6-8223-90d1d255334d-metrics-certs\") pod \"maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b\" (UID: \"f014fed4-d5bf-4ce6-8223-90d1d255334d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b" Nov 04 03:35:43.637928 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:43.637868 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6lv6\" (UniqueName: \"kubernetes.io/projected/f014fed4-d5bf-4ce6-8223-90d1d255334d-kube-api-access-c6lv6\") pod \"maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b\" (UID: \"f014fed4-d5bf-4ce6-8223-90d1d255334d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b" Nov 04 03:35:43.637928 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:35:43.637874 5953 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-d1e2ab-serving-cert: secret "maas-api-e2e-extmod-b-d1e2ab-serving-cert" not found Nov 04 03:35:43.638163 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:35:43.637958 5953 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f014fed4-d5bf-4ce6-8223-90d1d255334d-maas-api-tls podName:f014fed4-d5bf-4ce6-8223-90d1d255334d nodeName:}" failed. No retries permitted until 2026-11-04 03:35:44.137933827 +0000 UTC m=+1069.412564574 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f014fed4-d5bf-4ce6-8223-90d1d255334d-maas-api-tls") pod "maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b" (UID: "f014fed4-d5bf-4ce6-8223-90d1d255334d") : secret "maas-api-e2e-extmod-b-d1e2ab-serving-cert" not found Nov 04 03:35:43.638551 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:43.638521 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f014fed4-d5bf-4ce6-8223-90d1d255334d-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b\" (UID: \"f014fed4-d5bf-4ce6-8223-90d1d255334d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b" Nov 04 03:35:43.640900 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:43.640872 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f014fed4-d5bf-4ce6-8223-90d1d255334d-metrics-certs\") pod \"maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b\" (UID: \"f014fed4-d5bf-4ce6-8223-90d1d255334d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b" Nov 04 03:35:43.700112 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:43.700081 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6lv6\" (UniqueName: \"kubernetes.io/projected/f014fed4-d5bf-4ce6-8223-90d1d255334d-kube-api-access-c6lv6\") pod \"maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b\" (UID: \"f014fed4-d5bf-4ce6-8223-90d1d255334d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b" Nov 04 03:35:44.146520 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:44.146466 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f014fed4-d5bf-4ce6-8223-90d1d255334d-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b\" (UID: \"f014fed4-d5bf-4ce6-8223-90d1d255334d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b" Nov 04 03:35:44.149115 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:44.149090 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f014fed4-d5bf-4ce6-8223-90d1d255334d-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b\" (UID: \"f014fed4-d5bf-4ce6-8223-90d1d255334d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b" Nov 04 03:35:44.383058 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:44.383027 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b" Nov 04 03:35:44.768530 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:44.768439 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b"] Nov 04 03:35:44.771183 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:35:44.771148 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf014fed4_d5bf_4ce6_8223_90d1d255334d.slice/crio-c958dfb56091107c25130a26d38c75b3250ceae60e068dd82eef8c6f0fc506a7 WatchSource:0}: Error finding container c958dfb56091107c25130a26d38c75b3250ceae60e068dd82eef8c6f0fc506a7: Status 404 returned error can't find the container with id c958dfb56091107c25130a26d38c75b3250ceae60e068dd82eef8c6f0fc506a7 Nov 04 03:35:45.733245 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:45.733207 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b" event={"ID":"f014fed4-d5bf-4ce6-8223-90d1d255334d","Type":"ContainerStarted","Data":"5ca2fc5d7b362471e3316cbd1e89c7b98e72d57bb638a405d7a06f0f028e4e6d"} Nov 04 03:35:45.733407 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:45.733266 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b" event={"ID":"f014fed4-d5bf-4ce6-8223-90d1d255334d","Type":"ContainerStarted","Data":"c958dfb56091107c25130a26d38c75b3250ceae60e068dd82eef8c6f0fc506a7"} Nov 04 03:35:45.733407 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:45.733298 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b" Nov 04 03:35:45.852965 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:45.852912 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b" podStartSLOduration=2.008651164 podStartE2EDuration="2.852898488s" totalImagesPullingTime="844.247324ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:35:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:35:45.850555532 +0000 UTC m=+1071.125186306" watchObservedRunningTime="2026-11-04 03:35:45.852898488 +0000 UTC m=+1071.127529295" Nov 04 03:35:46.386087 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:35:46.386048 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:35:48.721371 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:48.721337 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g" Nov 04 03:35:51.746872 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:51.746844 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b" Nov 04 03:35:57.078194 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:57.078152 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8"] Nov 04 03:35:57.084294 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:57.084271 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" Nov 04 03:35:57.106843 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:57.106812 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be9f6d1-586c-493a-aa67-76e09cc64ff9-credential-socket\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8\" (UID: \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" Nov 04 03:35:57.107014 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:57.106852 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be9f6d1-586c-493a-aa67-76e09cc64ff9-istiod-ca-cert\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8\" (UID: \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" Nov 04 03:35:57.107014 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:57.106880 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm4l6\" (UniqueName: \"kubernetes.io/projected/4be9f6d1-586c-493a-aa67-76e09cc64ff9-kube-api-access-nm4l6\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8\" (UID: \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" Nov 04 03:35:57.107014 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:57.106964 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be9f6d1-586c-493a-aa67-76e09cc64ff9-workload-certs\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8\" (UID: \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" Nov 04 03:35:57.107203 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:57.107080 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be9f6d1-586c-493a-aa67-76e09cc64ff9-istio-podinfo\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8\" (UID: \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" Nov 04 03:35:57.107203 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:57.107132 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be9f6d1-586c-493a-aa67-76e09cc64ff9-istio-envoy\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8\" (UID: \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" Nov 04 03:35:57.107203 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:57.107170 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4be9f6d1-586c-493a-aa67-76e09cc64ff9-istio-token\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8\" (UID: \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" Nov 04 03:35:57.107203 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:57.107192 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be9f6d1-586c-493a-aa67-76e09cc64ff9-istio-data\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8\" (UID: \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" Nov 04 03:35:57.107468 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:57.107239 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be9f6d1-586c-493a-aa67-76e09cc64ff9-workload-socket\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8\" (UID: \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" Nov 04 03:35:57.137281 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:57.137242 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-af788a-openshift-default-dockercfg-826jt\"" Nov 04 03:35:57.191170 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:57.191134 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8"] Nov 04 03:35:57.208398 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:57.208363 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be9f6d1-586c-493a-aa67-76e09cc64ff9-credential-socket\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8\" (UID: \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" Nov 04 03:35:57.208398 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:57.208401 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be9f6d1-586c-493a-aa67-76e09cc64ff9-istiod-ca-cert\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8\" (UID: \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" Nov 04 03:35:57.208712 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:57.208421 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm4l6\" (UniqueName: \"kubernetes.io/projected/4be9f6d1-586c-493a-aa67-76e09cc64ff9-kube-api-access-nm4l6\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8\" (UID: \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" Nov 04 03:35:57.208712 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:57.208511 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be9f6d1-586c-493a-aa67-76e09cc64ff9-workload-certs\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8\" (UID: \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" Nov 04 03:35:57.208712 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:57.208564 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be9f6d1-586c-493a-aa67-76e09cc64ff9-istio-podinfo\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8\" (UID: \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" Nov 04 03:35:57.208712 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:57.208610 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be9f6d1-586c-493a-aa67-76e09cc64ff9-istio-envoy\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8\" (UID: \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" Nov 04 03:35:57.208712 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:57.208698 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4be9f6d1-586c-493a-aa67-76e09cc64ff9-istio-token\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8\" (UID: \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" Nov 04 03:35:57.208985 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:57.208727 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be9f6d1-586c-493a-aa67-76e09cc64ff9-istio-data\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8\" (UID: \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" Nov 04 03:35:57.208985 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:57.208791 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be9f6d1-586c-493a-aa67-76e09cc64ff9-workload-socket\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8\" (UID: \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" Nov 04 03:35:57.208985 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:57.208844 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be9f6d1-586c-493a-aa67-76e09cc64ff9-credential-socket\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8\" (UID: \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" Nov 04 03:35:57.209142 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:57.209093 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be9f6d1-586c-493a-aa67-76e09cc64ff9-workload-certs\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8\" (UID: \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" Nov 04 03:35:57.209142 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:57.209112 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be9f6d1-586c-493a-aa67-76e09cc64ff9-workload-socket\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8\" (UID: \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" Nov 04 03:35:57.209307 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:57.209281 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be9f6d1-586c-493a-aa67-76e09cc64ff9-istiod-ca-cert\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8\" (UID: \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" Nov 04 03:35:57.209410 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:57.209392 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be9f6d1-586c-493a-aa67-76e09cc64ff9-istio-data\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8\" (UID: \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" Nov 04 03:35:57.215402 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:57.215156 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be9f6d1-586c-493a-aa67-76e09cc64ff9-istio-podinfo\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8\" (UID: \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" Nov 04 03:35:57.215402 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:57.215351 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be9f6d1-586c-493a-aa67-76e09cc64ff9-istio-envoy\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8\" (UID: \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" Nov 04 03:35:57.270651 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:57.270506 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4be9f6d1-586c-493a-aa67-76e09cc64ff9-istio-token\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8\" (UID: \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" Nov 04 03:35:57.271101 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:57.271073 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm4l6\" (UniqueName: \"kubernetes.io/projected/4be9f6d1-586c-493a-aa67-76e09cc64ff9-kube-api-access-nm4l6\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8\" (UID: \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" Nov 04 03:35:57.402056 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:57.401972 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" Nov 04 03:35:57.848361 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:57.848336 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8"] Nov 04 03:35:57.849888 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:35:57.849855 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4be9f6d1_586c_493a_aa67_76e09cc64ff9.slice/crio-85def7e19d3d28b3684d26c073343e42b8e89b511278aa5e9c15a1a3fc704426 WatchSource:0}: Error finding container 85def7e19d3d28b3684d26c073343e42b8e89b511278aa5e9c15a1a3fc704426: Status 404 returned error can't find the container with id 85def7e19d3d28b3684d26c073343e42b8e89b511278aa5e9c15a1a3fc704426 Nov 04 03:35:57.852843 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:57.852803 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:35:57.852921 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:57.852898 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:35:57.852959 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:57.852942 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:35:58.835070 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:58.835035 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" event={"ID":"4be9f6d1-586c-493a-aa67-76e09cc64ff9","Type":"ContainerStarted","Data":"61fc1a76802e443bc2ae45f6efeb3a996f4a116056e663049a266d243cbe35df"} Nov 04 03:35:58.835539 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:58.835112 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" event={"ID":"4be9f6d1-586c-493a-aa67-76e09cc64ff9","Type":"ContainerStarted","Data":"85def7e19d3d28b3684d26c073343e42b8e89b511278aa5e9c15a1a3fc704426"} Nov 04 03:35:59.025644 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:59.025046 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" podStartSLOduration=2.025012449 podStartE2EDuration="2.025012449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:35:59.023966241 +0000 UTC m=+1084.298597017" watchObservedRunningTime="2026-11-04 03:35:59.025012449 +0000 UTC m=+1084.299643213" Nov 04 03:35:59.427842 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:59.427807 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" Nov 04 03:35:59.429312 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:59.429280 5953 patch_prober.go:29] interesting pod/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 04 03:35:59.429469 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:35:59.429339 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" podUID="4be9f6d1-586c-493a-aa67-76e09cc64ff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 04 03:36:00.414646 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:00.406905 5953 patch_prober.go:29] interesting pod/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 04 03:36:00.414646 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:00.406963 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" podUID="4be9f6d1-586c-493a-aa67-76e09cc64ff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 04 03:36:01.412988 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:01.412951 5953 patch_prober.go:29] interesting pod/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 04 03:36:01.413264 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:01.413236 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" podUID="4be9f6d1-586c-493a-aa67-76e09cc64ff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 04 03:36:02.402514 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:02.402482 5953 patch_prober.go:29] interesting pod/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 04 03:36:02.403064 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:02.402533 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" podUID="4be9f6d1-586c-493a-aa67-76e09cc64ff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 04 03:36:02.994760 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:02.994728 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n"] Nov 04 03:36:02.999146 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:02.999123 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n" Nov 04 03:36:03.054803 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:03.054774 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-491fe7-serving-cert\"" Nov 04 03:36:03.057602 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:03.057575 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n"] Nov 04 03:36:03.090559 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:03.090530 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71a3edb7-8071-4977-9977-4adfcceebdcf-maas-api-tls\") pod \"maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n\" (UID: \"71a3edb7-8071-4977-9977-4adfcceebdcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n" Nov 04 03:36:03.090719 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:03.090604 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jzm6\" (UniqueName: \"kubernetes.io/projected/71a3edb7-8071-4977-9977-4adfcceebdcf-kube-api-access-4jzm6\") pod \"maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n\" (UID: \"71a3edb7-8071-4977-9977-4adfcceebdcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n" Nov 04 03:36:03.090778 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:03.090721 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71a3edb7-8071-4977-9977-4adfcceebdcf-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n\" (UID: \"71a3edb7-8071-4977-9977-4adfcceebdcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n" Nov 04 03:36:03.090778 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:03.090765 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71a3edb7-8071-4977-9977-4adfcceebdcf-metrics-certs\") pod \"maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n\" (UID: \"71a3edb7-8071-4977-9977-4adfcceebdcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n" Nov 04 03:36:03.194202 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:03.193298 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71a3edb7-8071-4977-9977-4adfcceebdcf-maas-api-tls\") pod \"maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n\" (UID: \"71a3edb7-8071-4977-9977-4adfcceebdcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n" Nov 04 03:36:03.194202 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:03.193410 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jzm6\" (UniqueName: \"kubernetes.io/projected/71a3edb7-8071-4977-9977-4adfcceebdcf-kube-api-access-4jzm6\") pod \"maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n\" (UID: \"71a3edb7-8071-4977-9977-4adfcceebdcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n" Nov 04 03:36:03.194202 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:03.193507 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71a3edb7-8071-4977-9977-4adfcceebdcf-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n\" (UID: \"71a3edb7-8071-4977-9977-4adfcceebdcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n" Nov 04 03:36:03.194202 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:03.193575 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71a3edb7-8071-4977-9977-4adfcceebdcf-metrics-certs\") pod \"maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n\" (UID: \"71a3edb7-8071-4977-9977-4adfcceebdcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n" Nov 04 03:36:03.197613 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:36:03.196226 5953 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-491fe7-serving-cert: secret "maas-api-e2e-extmod-b-491fe7-serving-cert" not found Nov 04 03:36:03.197613 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:36:03.196304 5953 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71a3edb7-8071-4977-9977-4adfcceebdcf-maas-api-tls podName:71a3edb7-8071-4977-9977-4adfcceebdcf nodeName:}" failed. No retries permitted until 2026-11-04 03:36:03.696285334 +0000 UTC m=+1088.970916089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/71a3edb7-8071-4977-9977-4adfcceebdcf-maas-api-tls") pod "maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n" (UID: "71a3edb7-8071-4977-9977-4adfcceebdcf") : secret "maas-api-e2e-extmod-b-491fe7-serving-cert" not found Nov 04 03:36:03.197613 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:03.197560 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71a3edb7-8071-4977-9977-4adfcceebdcf-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n\" (UID: \"71a3edb7-8071-4977-9977-4adfcceebdcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n" Nov 04 03:36:03.200561 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:03.200493 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71a3edb7-8071-4977-9977-4adfcceebdcf-metrics-certs\") pod \"maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n\" (UID: \"71a3edb7-8071-4977-9977-4adfcceebdcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n" Nov 04 03:36:03.263250 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:03.263202 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jzm6\" (UniqueName: \"kubernetes.io/projected/71a3edb7-8071-4977-9977-4adfcceebdcf-kube-api-access-4jzm6\") pod \"maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n\" (UID: \"71a3edb7-8071-4977-9977-4adfcceebdcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n" Nov 04 03:36:03.402410 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:03.402379 5953 patch_prober.go:29] interesting pod/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 04 03:36:03.402586 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:03.402430 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" podUID="4be9f6d1-586c-493a-aa67-76e09cc64ff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 04 03:36:03.701069 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:03.700954 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71a3edb7-8071-4977-9977-4adfcceebdcf-maas-api-tls\") pod \"maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n\" (UID: \"71a3edb7-8071-4977-9977-4adfcceebdcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n" Nov 04 03:36:03.732012 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:03.731977 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71a3edb7-8071-4977-9977-4adfcceebdcf-maas-api-tls\") pod \"maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n\" (UID: \"71a3edb7-8071-4977-9977-4adfcceebdcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n" Nov 04 03:36:03.934333 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:03.934299 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n" Nov 04 03:36:04.316873 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:04.315951 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n"] Nov 04 03:36:04.329480 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:36:04.329435 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71a3edb7_8071_4977_9977_4adfcceebdcf.slice/crio-b851a92b5d84b93a95da164b38fba24e3e26737d7a6b19f052e3ac0427093a9f WatchSource:0}: Error finding container b851a92b5d84b93a95da164b38fba24e3e26737d7a6b19f052e3ac0427093a9f: Status 404 returned error can't find the container with id b851a92b5d84b93a95da164b38fba24e3e26737d7a6b19f052e3ac0427093a9f Nov 04 03:36:04.403158 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:04.403127 5953 patch_prober.go:29] interesting pod/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 04 03:36:04.403665 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:04.403183 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" podUID="4be9f6d1-586c-493a-aa67-76e09cc64ff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 04 03:36:04.874744 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:04.874707 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n" event={"ID":"71a3edb7-8071-4977-9977-4adfcceebdcf","Type":"ContainerStarted","Data":"b851a92b5d84b93a95da164b38fba24e3e26737d7a6b19f052e3ac0427093a9f"} Nov 04 03:36:05.406750 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:05.406718 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" Nov 04 03:36:05.407141 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:05.406984 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" Nov 04 03:36:05.407680 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:05.407654 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" Nov 04 03:36:05.882535 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:05.882497 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n" event={"ID":"71a3edb7-8071-4977-9977-4adfcceebdcf","Type":"ContainerStarted","Data":"b5c3b92e45949e19fa4c37408c1751327ff415e6006d5501c6a1a374a9a1fe2a"} Nov 04 03:36:06.001737 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:06.001686 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n" podStartSLOduration=3.130933126 podStartE2EDuration="4.001670752s" totalImagesPullingTime="870.737626ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:36:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:36:06.000818618 +0000 UTC m=+1091.275449380" watchObservedRunningTime="2026-11-04 03:36:06.001670752 +0000 UTC m=+1091.276301515" Nov 04 03:36:06.888042 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:06.888010 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n" Nov 04 03:36:07.685629 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:07.685596 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc"] Nov 04 03:36:07.689725 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:07.689704 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" Nov 04 03:36:07.744657 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:07.744614 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-cr7gn\"" Nov 04 03:36:07.749127 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:07.749103 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75dde092-12d7-4855-a6ba-3edb9e650a5c-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-2nvtc\" (UID: \"75dde092-12d7-4855-a6ba-3edb9e650a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" Nov 04 03:36:07.749258 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:07.749135 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75dde092-12d7-4855-a6ba-3edb9e650a5c-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-2nvtc\" (UID: \"75dde092-12d7-4855-a6ba-3edb9e650a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" Nov 04 03:36:07.749258 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:07.749158 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75dde092-12d7-4855-a6ba-3edb9e650a5c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-2nvtc\" (UID: \"75dde092-12d7-4855-a6ba-3edb9e650a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" Nov 04 03:36:07.749342 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:07.749277 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75dde092-12d7-4855-a6ba-3edb9e650a5c-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-2nvtc\" (UID: \"75dde092-12d7-4855-a6ba-3edb9e650a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" Nov 04 03:36:07.749382 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:07.749368 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g97g\" (UniqueName: \"kubernetes.io/projected/75dde092-12d7-4855-a6ba-3edb9e650a5c-kube-api-access-5g97g\") pod \"maas-default-gateway-openshift-default-6dcf757899-2nvtc\" (UID: \"75dde092-12d7-4855-a6ba-3edb9e650a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" Nov 04 03:36:07.749442 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:07.749429 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75dde092-12d7-4855-a6ba-3edb9e650a5c-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-2nvtc\" (UID: \"75dde092-12d7-4855-a6ba-3edb9e650a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" Nov 04 03:36:07.749503 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:07.749489 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75dde092-12d7-4855-a6ba-3edb9e650a5c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-2nvtc\" (UID: \"75dde092-12d7-4855-a6ba-3edb9e650a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" Nov 04 03:36:07.749586 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:07.749571 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75dde092-12d7-4855-a6ba-3edb9e650a5c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-2nvtc\" (UID: \"75dde092-12d7-4855-a6ba-3edb9e650a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" Nov 04 03:36:07.749654 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:07.749593 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75dde092-12d7-4855-a6ba-3edb9e650a5c-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-2nvtc\" (UID: \"75dde092-12d7-4855-a6ba-3edb9e650a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" Nov 04 03:36:07.797612 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:07.797584 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc"] Nov 04 03:36:07.851155 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:07.851124 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75dde092-12d7-4855-a6ba-3edb9e650a5c-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-2nvtc\" (UID: \"75dde092-12d7-4855-a6ba-3edb9e650a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" Nov 04 03:36:07.851332 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:07.851206 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g97g\" (UniqueName: \"kubernetes.io/projected/75dde092-12d7-4855-a6ba-3edb9e650a5c-kube-api-access-5g97g\") pod \"maas-default-gateway-openshift-default-6dcf757899-2nvtc\" (UID: \"75dde092-12d7-4855-a6ba-3edb9e650a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" Nov 04 03:36:07.851332 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:07.851268 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75dde092-12d7-4855-a6ba-3edb9e650a5c-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-2nvtc\" (UID: \"75dde092-12d7-4855-a6ba-3edb9e650a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" Nov 04 03:36:07.851332 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:07.851322 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75dde092-12d7-4855-a6ba-3edb9e650a5c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-2nvtc\" (UID: \"75dde092-12d7-4855-a6ba-3edb9e650a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" Nov 04 03:36:07.851528 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:07.851400 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75dde092-12d7-4855-a6ba-3edb9e650a5c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-2nvtc\" (UID: \"75dde092-12d7-4855-a6ba-3edb9e650a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" Nov 04 03:36:07.851528 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:07.851435 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75dde092-12d7-4855-a6ba-3edb9e650a5c-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-2nvtc\" (UID: \"75dde092-12d7-4855-a6ba-3edb9e650a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" Nov 04 03:36:07.851528 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:07.851447 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75dde092-12d7-4855-a6ba-3edb9e650a5c-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-2nvtc\" (UID: \"75dde092-12d7-4855-a6ba-3edb9e650a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" Nov 04 03:36:07.851528 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:07.851478 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75dde092-12d7-4855-a6ba-3edb9e650a5c-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-2nvtc\" (UID: \"75dde092-12d7-4855-a6ba-3edb9e650a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" Nov 04 03:36:07.851528 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:07.851507 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75dde092-12d7-4855-a6ba-3edb9e650a5c-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-2nvtc\" (UID: \"75dde092-12d7-4855-a6ba-3edb9e650a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" Nov 04 03:36:07.851826 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:07.851555 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75dde092-12d7-4855-a6ba-3edb9e650a5c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-2nvtc\" (UID: \"75dde092-12d7-4855-a6ba-3edb9e650a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" Nov 04 03:36:07.851826 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:07.851659 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75dde092-12d7-4855-a6ba-3edb9e650a5c-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-2nvtc\" (UID: \"75dde092-12d7-4855-a6ba-3edb9e650a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" Nov 04 03:36:07.851932 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:07.851828 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75dde092-12d7-4855-a6ba-3edb9e650a5c-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-2nvtc\" (UID: \"75dde092-12d7-4855-a6ba-3edb9e650a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" Nov 04 03:36:07.851932 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:07.851916 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75dde092-12d7-4855-a6ba-3edb9e650a5c-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-2nvtc\" (UID: \"75dde092-12d7-4855-a6ba-3edb9e650a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" Nov 04 03:36:07.852052 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:07.852035 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75dde092-12d7-4855-a6ba-3edb9e650a5c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-2nvtc\" (UID: \"75dde092-12d7-4855-a6ba-3edb9e650a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" Nov 04 03:36:07.853827 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:07.853809 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75dde092-12d7-4855-a6ba-3edb9e650a5c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-2nvtc\" (UID: \"75dde092-12d7-4855-a6ba-3edb9e650a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" Nov 04 03:36:07.853927 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:07.853876 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75dde092-12d7-4855-a6ba-3edb9e650a5c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-2nvtc\" (UID: \"75dde092-12d7-4855-a6ba-3edb9e650a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" Nov 04 03:36:07.914044 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:07.914018 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75dde092-12d7-4855-a6ba-3edb9e650a5c-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-2nvtc\" (UID: \"75dde092-12d7-4855-a6ba-3edb9e650a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" Nov 04 03:36:07.914480 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:07.914203 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g97g\" (UniqueName: \"kubernetes.io/projected/75dde092-12d7-4855-a6ba-3edb9e650a5c-kube-api-access-5g97g\") pod \"maas-default-gateway-openshift-default-6dcf757899-2nvtc\" (UID: \"75dde092-12d7-4855-a6ba-3edb9e650a5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" Nov 04 03:36:08.003672 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:08.003637 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" Nov 04 03:36:08.431007 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:08.430984 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc"] Nov 04 03:36:08.433203 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:36:08.433166 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75dde092_12d7_4855_a6ba_3edb9e650a5c.slice/crio-e324930e27662f499f8fcb63cdde72ab5341c228c4659eb006c2f338131147d8 WatchSource:0}: Error finding container e324930e27662f499f8fcb63cdde72ab5341c228c4659eb006c2f338131147d8: Status 404 returned error can't find the container with id e324930e27662f499f8fcb63cdde72ab5341c228c4659eb006c2f338131147d8 Nov 04 03:36:08.435641 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:08.435587 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:36:08.435750 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:08.435666 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:36:08.435750 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:08.435698 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:36:08.904966 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:08.904933 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" event={"ID":"75dde092-12d7-4855-a6ba-3edb9e650a5c","Type":"ContainerStarted","Data":"698f53220765bf3690d6d35ce7ccb78d6e02a804a4d9b4969a07478d8287e3e7"} Nov 04 03:36:08.905128 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:08.904987 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" event={"ID":"75dde092-12d7-4855-a6ba-3edb9e650a5c","Type":"ContainerStarted","Data":"e324930e27662f499f8fcb63cdde72ab5341c228c4659eb006c2f338131147d8"} Nov 04 03:36:09.018049 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:09.018013 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" Nov 04 03:36:09.021155 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:09.021126 5953 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-2nvtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 04 03:36:09.021363 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:09.021338 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" podUID="75dde092-12d7-4855-a6ba-3edb9e650a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 04 03:36:09.031793 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:09.031643 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" podStartSLOduration=2.031605681 podStartE2EDuration="2.031605681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:36:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:36:09.030050265 +0000 UTC m=+1094.304681027" watchObservedRunningTime="2026-11-04 03:36:09.031605681 +0000 UTC m=+1094.306236447" Nov 04 03:36:10.028062 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:10.028021 5953 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-2nvtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 04 03:36:10.028566 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:10.028089 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" podUID="75dde092-12d7-4855-a6ba-3edb9e650a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 04 03:36:11.012480 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:11.012025 5953 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-2nvtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 04 03:36:11.012480 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:11.012098 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" podUID="75dde092-12d7-4855-a6ba-3edb9e650a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 04 03:36:12.028127 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:12.028093 5953 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-2nvtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 04 03:36:12.028607 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:12.028175 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" podUID="75dde092-12d7-4855-a6ba-3edb9e650a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 04 03:36:12.901885 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:12.901856 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n" Nov 04 03:36:13.015363 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:13.015327 5953 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-2nvtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 04 03:36:13.015674 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:13.015640 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" podUID="75dde092-12d7-4855-a6ba-3edb9e650a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 04 03:36:14.018064 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:14.018025 5953 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-2nvtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 04 03:36:14.018688 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:14.018653 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" podUID="75dde092-12d7-4855-a6ba-3edb9e650a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 04 03:36:15.004430 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:15.004398 5953 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-2nvtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 04 03:36:15.004609 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:15.004455 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" podUID="75dde092-12d7-4855-a6ba-3edb9e650a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 04 03:36:16.004879 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:16.004846 5953 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-2nvtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 04 03:36:16.005271 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:16.004909 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" podUID="75dde092-12d7-4855-a6ba-3edb9e650a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 04 03:36:16.385570 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:36:16.385474 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:36:17.179386 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:17.179342 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" Nov 04 03:36:17.180173 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:17.180137 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" Nov 04 03:36:17.181415 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:17.181392 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2nvtc" Nov 04 03:36:22.840904 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:22.840863 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4"] Nov 04 03:36:22.845661 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:22.845639 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" Nov 04 03:36:22.867801 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:22.867745 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c16e30a6-2c91-4687-91ff-6b496c753277-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-rmgs4\" (UID: \"c16e30a6-2c91-4687-91ff-6b496c753277\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" Nov 04 03:36:22.867982 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:22.867868 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c16e30a6-2c91-4687-91ff-6b496c753277-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-rmgs4\" (UID: \"c16e30a6-2c91-4687-91ff-6b496c753277\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" Nov 04 03:36:22.867982 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:22.867903 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16e30a6-2c91-4687-91ff-6b496c753277-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-rmgs4\" (UID: \"c16e30a6-2c91-4687-91ff-6b496c753277\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" Nov 04 03:36:22.867982 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:22.867945 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16e30a6-2c91-4687-91ff-6b496c753277-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-rmgs4\" (UID: \"c16e30a6-2c91-4687-91ff-6b496c753277\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" Nov 04 03:36:22.868150 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:22.867996 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c16e30a6-2c91-4687-91ff-6b496c753277-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-rmgs4\" (UID: \"c16e30a6-2c91-4687-91ff-6b496c753277\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" Nov 04 03:36:22.868150 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:22.868037 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c16e30a6-2c91-4687-91ff-6b496c753277-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-rmgs4\" (UID: \"c16e30a6-2c91-4687-91ff-6b496c753277\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" Nov 04 03:36:22.868150 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:22.868075 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c16e30a6-2c91-4687-91ff-6b496c753277-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-rmgs4\" (UID: \"c16e30a6-2c91-4687-91ff-6b496c753277\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" Nov 04 03:36:22.868150 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:22.868125 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sldxq\" (UniqueName: \"kubernetes.io/projected/c16e30a6-2c91-4687-91ff-6b496c753277-kube-api-access-sldxq\") pod \"maas-default-gateway-openshift-default-6dcf757899-rmgs4\" (UID: \"c16e30a6-2c91-4687-91ff-6b496c753277\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" Nov 04 03:36:22.868358 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:22.868185 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c16e30a6-2c91-4687-91ff-6b496c753277-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-rmgs4\" (UID: \"c16e30a6-2c91-4687-91ff-6b496c753277\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" Nov 04 03:36:22.969076 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:22.969044 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16e30a6-2c91-4687-91ff-6b496c753277-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-rmgs4\" (UID: \"c16e30a6-2c91-4687-91ff-6b496c753277\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" Nov 04 03:36:22.969275 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:22.969115 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c16e30a6-2c91-4687-91ff-6b496c753277-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-rmgs4\" (UID: \"c16e30a6-2c91-4687-91ff-6b496c753277\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" Nov 04 03:36:22.969275 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:22.969143 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c16e30a6-2c91-4687-91ff-6b496c753277-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-rmgs4\" (UID: \"c16e30a6-2c91-4687-91ff-6b496c753277\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" Nov 04 03:36:22.969275 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:22.969180 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c16e30a6-2c91-4687-91ff-6b496c753277-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-rmgs4\" (UID: \"c16e30a6-2c91-4687-91ff-6b496c753277\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" Nov 04 03:36:22.969275 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:22.969227 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sldxq\" (UniqueName: \"kubernetes.io/projected/c16e30a6-2c91-4687-91ff-6b496c753277-kube-api-access-sldxq\") pod \"maas-default-gateway-openshift-default-6dcf757899-rmgs4\" (UID: \"c16e30a6-2c91-4687-91ff-6b496c753277\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" Nov 04 03:36:22.969478 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:22.969280 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c16e30a6-2c91-4687-91ff-6b496c753277-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-rmgs4\" (UID: \"c16e30a6-2c91-4687-91ff-6b496c753277\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" Nov 04 03:36:22.969478 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:22.969363 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c16e30a6-2c91-4687-91ff-6b496c753277-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-rmgs4\" (UID: \"c16e30a6-2c91-4687-91ff-6b496c753277\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" Nov 04 03:36:22.969578 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:22.969479 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c16e30a6-2c91-4687-91ff-6b496c753277-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-rmgs4\" (UID: \"c16e30a6-2c91-4687-91ff-6b496c753277\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" Nov 04 03:36:22.969578 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:22.969518 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16e30a6-2c91-4687-91ff-6b496c753277-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-rmgs4\" (UID: \"c16e30a6-2c91-4687-91ff-6b496c753277\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" Nov 04 03:36:22.969922 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:22.969900 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16e30a6-2c91-4687-91ff-6b496c753277-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-rmgs4\" (UID: \"c16e30a6-2c91-4687-91ff-6b496c753277\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" Nov 04 03:36:22.970460 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:22.970426 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c16e30a6-2c91-4687-91ff-6b496c753277-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-rmgs4\" (UID: \"c16e30a6-2c91-4687-91ff-6b496c753277\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" Nov 04 03:36:22.970579 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:22.970517 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c16e30a6-2c91-4687-91ff-6b496c753277-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-rmgs4\" (UID: \"c16e30a6-2c91-4687-91ff-6b496c753277\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" Nov 04 03:36:22.971178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:22.971155 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16e30a6-2c91-4687-91ff-6b496c753277-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-rmgs4\" (UID: \"c16e30a6-2c91-4687-91ff-6b496c753277\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" Nov 04 03:36:22.972028 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:22.972003 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c16e30a6-2c91-4687-91ff-6b496c753277-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-rmgs4\" (UID: \"c16e30a6-2c91-4687-91ff-6b496c753277\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" Nov 04 03:36:22.974543 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:22.973260 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c16e30a6-2c91-4687-91ff-6b496c753277-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-rmgs4\" (UID: \"c16e30a6-2c91-4687-91ff-6b496c753277\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" Nov 04 03:36:22.978028 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:22.978004 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c16e30a6-2c91-4687-91ff-6b496c753277-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-rmgs4\" (UID: \"c16e30a6-2c91-4687-91ff-6b496c753277\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" Nov 04 03:36:23.031976 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:23.031944 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sldxq\" (UniqueName: \"kubernetes.io/projected/c16e30a6-2c91-4687-91ff-6b496c753277-kube-api-access-sldxq\") pod \"maas-default-gateway-openshift-default-6dcf757899-rmgs4\" (UID: \"c16e30a6-2c91-4687-91ff-6b496c753277\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" Nov 04 03:36:23.031976 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:23.031969 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c16e30a6-2c91-4687-91ff-6b496c753277-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-rmgs4\" (UID: \"c16e30a6-2c91-4687-91ff-6b496c753277\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" Nov 04 03:36:23.053394 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:23.053360 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4"] Nov 04 03:36:23.167691 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:23.167572 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" Nov 04 03:36:23.614388 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:23.613526 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4"] Nov 04 03:36:23.616680 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:36:23.616595 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc16e30a6_2c91_4687_91ff_6b496c753277.slice/crio-29d553525519f450327a92febe582d873c17291272e3b7e414329289813704ce WatchSource:0}: Error finding container 29d553525519f450327a92febe582d873c17291272e3b7e414329289813704ce: Status 404 returned error can't find the container with id 29d553525519f450327a92febe582d873c17291272e3b7e414329289813704ce Nov 04 03:36:23.619454 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:23.619419 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:36:23.619562 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:23.619490 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:36:23.619562 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:23.619519 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:36:24.077169 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:24.077131 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" event={"ID":"c16e30a6-2c91-4687-91ff-6b496c753277","Type":"ContainerStarted","Data":"78fa81312d71542ed47724e499e7a09a097e6b1c2809350505c6d471efed13ea"} Nov 04 03:36:24.077573 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:24.077185 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" event={"ID":"c16e30a6-2c91-4687-91ff-6b496c753277","Type":"ContainerStarted","Data":"29d553525519f450327a92febe582d873c17291272e3b7e414329289813704ce"} Nov 04 03:36:24.168115 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:24.168057 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" Nov 04 03:36:24.170199 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:24.170168 5953 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-rmgs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 04 03:36:24.170358 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:24.170229 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" podUID="c16e30a6-2c91-4687-91ff-6b496c753277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 04 03:36:24.278836 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:24.278784 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" podStartSLOduration=2.278769386 podStartE2EDuration="2.278769386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:36:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:36:24.250747762 +0000 UTC m=+1109.525378524" watchObservedRunningTime="2026-11-04 03:36:24.278769386 +0000 UTC m=+1109.553400185" Nov 04 03:36:25.168150 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:25.168057 5953 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-rmgs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 04 03:36:25.168687 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:25.168227 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" podUID="c16e30a6-2c91-4687-91ff-6b496c753277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 04 03:36:26.168272 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:26.168234 5953 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-rmgs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 04 03:36:26.168782 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:26.168289 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" podUID="c16e30a6-2c91-4687-91ff-6b496c753277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 04 03:36:27.168807 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:27.168768 5953 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-rmgs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 04 03:36:27.169316 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:27.168824 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" podUID="c16e30a6-2c91-4687-91ff-6b496c753277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 04 03:36:28.168974 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:28.168940 5953 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-rmgs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 04 03:36:28.169488 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:28.168999 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" podUID="c16e30a6-2c91-4687-91ff-6b496c753277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 04 03:36:29.168275 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:29.168236 5953 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-rmgs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 04 03:36:29.168466 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:29.168306 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" podUID="c16e30a6-2c91-4687-91ff-6b496c753277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 04 03:36:30.168813 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:30.168766 5953 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-rmgs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 04 03:36:30.169338 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:30.168815 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" podUID="c16e30a6-2c91-4687-91ff-6b496c753277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 04 03:36:31.168707 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:31.168664 5953 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-rmgs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 04 03:36:31.168945 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:31.168737 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" podUID="c16e30a6-2c91-4687-91ff-6b496c753277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 04 03:36:31.324056 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:31.324019 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b"] Nov 04 03:36:31.328831 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:31.328801 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" Nov 04 03:36:31.381916 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:31.381882 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-491fe7-openshift-default-dockercfg-fpm7q\"" Nov 04 03:36:31.393140 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:31.392025 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0835f6a-4fcd-48a0-9557-781942c5ddd3-workload-socket\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b\" (UID: \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" Nov 04 03:36:31.393140 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:31.392158 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0835f6a-4fcd-48a0-9557-781942c5ddd3-istio-podinfo\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b\" (UID: \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" Nov 04 03:36:31.393140 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:31.392294 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0835f6a-4fcd-48a0-9557-781942c5ddd3-istio-data\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b\" (UID: \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" Nov 04 03:36:31.393140 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:31.392511 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0835f6a-4fcd-48a0-9557-781942c5ddd3-workload-certs\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b\" (UID: \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" Nov 04 03:36:31.393140 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:31.392597 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0835f6a-4fcd-48a0-9557-781942c5ddd3-istio-token\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b\" (UID: \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" Nov 04 03:36:31.393140 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:31.392796 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0835f6a-4fcd-48a0-9557-781942c5ddd3-istio-envoy\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b\" (UID: \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" Nov 04 03:36:31.393140 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:31.392846 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0835f6a-4fcd-48a0-9557-781942c5ddd3-istiod-ca-cert\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b\" (UID: \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" Nov 04 03:36:31.393140 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:31.392906 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0835f6a-4fcd-48a0-9557-781942c5ddd3-credential-socket\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b\" (UID: \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" Nov 04 03:36:31.393140 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:31.392930 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwlg8\" (UniqueName: \"kubernetes.io/projected/d0835f6a-4fcd-48a0-9557-781942c5ddd3-kube-api-access-mwlg8\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b\" (UID: \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" Nov 04 03:36:31.436708 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:31.436597 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b"] Nov 04 03:36:31.494915 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:31.494860 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0835f6a-4fcd-48a0-9557-781942c5ddd3-istio-podinfo\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b\" (UID: \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" Nov 04 03:36:31.494915 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:31.494916 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0835f6a-4fcd-48a0-9557-781942c5ddd3-istio-data\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b\" (UID: \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" Nov 04 03:36:31.495199 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:31.494956 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0835f6a-4fcd-48a0-9557-781942c5ddd3-workload-certs\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b\" (UID: \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" Nov 04 03:36:31.495199 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:31.495011 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0835f6a-4fcd-48a0-9557-781942c5ddd3-istio-token\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b\" (UID: \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" Nov 04 03:36:31.495199 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:31.495099 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0835f6a-4fcd-48a0-9557-781942c5ddd3-istio-envoy\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b\" (UID: \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" Nov 04 03:36:31.495199 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:31.495140 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0835f6a-4fcd-48a0-9557-781942c5ddd3-istiod-ca-cert\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b\" (UID: \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" Nov 04 03:36:31.495199 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:31.495195 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0835f6a-4fcd-48a0-9557-781942c5ddd3-credential-socket\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b\" (UID: \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" Nov 04 03:36:31.495493 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:31.495221 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwlg8\" (UniqueName: \"kubernetes.io/projected/d0835f6a-4fcd-48a0-9557-781942c5ddd3-kube-api-access-mwlg8\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b\" (UID: \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" Nov 04 03:36:31.495493 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:31.495347 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0835f6a-4fcd-48a0-9557-781942c5ddd3-workload-socket\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b\" (UID: \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" Nov 04 03:36:31.496524 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:31.496492 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0835f6a-4fcd-48a0-9557-781942c5ddd3-istiod-ca-cert\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b\" (UID: \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" Nov 04 03:36:31.496721 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:31.496605 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0835f6a-4fcd-48a0-9557-781942c5ddd3-credential-socket\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b\" (UID: \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" Nov 04 03:36:31.496900 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:31.496876 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0835f6a-4fcd-48a0-9557-781942c5ddd3-workload-socket\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b\" (UID: \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" Nov 04 03:36:31.497340 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:31.497315 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0835f6a-4fcd-48a0-9557-781942c5ddd3-istio-data\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b\" (UID: \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" Nov 04 03:36:31.497517 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:31.497496 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0835f6a-4fcd-48a0-9557-781942c5ddd3-workload-certs\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b\" (UID: \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" Nov 04 03:36:31.498283 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:31.498256 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0835f6a-4fcd-48a0-9557-781942c5ddd3-istio-podinfo\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b\" (UID: \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" Nov 04 03:36:31.498807 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:31.498784 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0835f6a-4fcd-48a0-9557-781942c5ddd3-istio-envoy\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b\" (UID: \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" Nov 04 03:36:31.556763 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:31.556724 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0835f6a-4fcd-48a0-9557-781942c5ddd3-istio-token\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b\" (UID: \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" Nov 04 03:36:31.557666 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:31.557603 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwlg8\" (UniqueName: \"kubernetes.io/projected/d0835f6a-4fcd-48a0-9557-781942c5ddd3-kube-api-access-mwlg8\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b\" (UID: \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" Nov 04 03:36:31.645974 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:31.645936 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" Nov 04 03:36:32.089896 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:32.089868 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b"] Nov 04 03:36:32.093495 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:36:32.093464 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0835f6a_4fcd_48a0_9557_781942c5ddd3.slice/crio-0c26a5535e09f4d9466dd7932428ad960c003a17cb55151cc9431387cea62f5e WatchSource:0}: Error finding container 0c26a5535e09f4d9466dd7932428ad960c003a17cb55151cc9431387cea62f5e: Status 404 returned error can't find the container with id 0c26a5535e09f4d9466dd7932428ad960c003a17cb55151cc9431387cea62f5e Nov 04 03:36:32.096534 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:32.096498 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:36:32.096694 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:32.096589 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:36:32.096694 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:32.096671 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:36:32.163192 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:32.163095 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" event={"ID":"d0835f6a-4fcd-48a0-9557-781942c5ddd3","Type":"ContainerStarted","Data":"0c26a5535e09f4d9466dd7932428ad960c003a17cb55151cc9431387cea62f5e"} Nov 04 03:36:32.713585 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:32.713556 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" Nov 04 03:36:32.714052 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:32.713950 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" Nov 04 03:36:32.715274 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:32.715255 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-rmgs4" Nov 04 03:36:33.171992 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:33.171951 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" event={"ID":"d0835f6a-4fcd-48a0-9557-781942c5ddd3","Type":"ContainerStarted","Data":"79035cff882eee49fa0039a6627787c790732f69144db4d4208bb1e1d00f7850"} Nov 04 03:36:33.646969 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:33.646929 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" Nov 04 03:36:33.648791 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:33.648766 5953 patch_prober.go:29] interesting pod/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 04 03:36:33.648929 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:33.648815 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" podUID="d0835f6a-4fcd-48a0-9557-781942c5ddd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 04 03:36:34.647351 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:34.647318 5953 patch_prober.go:29] interesting pod/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 04 03:36:34.647800 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:34.647374 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" podUID="d0835f6a-4fcd-48a0-9557-781942c5ddd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 04 03:36:35.646512 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:35.646473 5953 patch_prober.go:29] interesting pod/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 04 03:36:35.646742 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:35.646535 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" podUID="d0835f6a-4fcd-48a0-9557-781942c5ddd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 04 03:36:36.647239 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:36.647204 5953 patch_prober.go:29] interesting pod/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 04 03:36:36.647667 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:36.647259 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" podUID="d0835f6a-4fcd-48a0-9557-781942c5ddd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 04 03:36:37.647034 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:37.646997 5953 patch_prober.go:29] interesting pod/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 04 03:36:37.647225 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:37.647051 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" podUID="d0835f6a-4fcd-48a0-9557-781942c5ddd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 04 03:36:38.646515 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:38.646483 5953 patch_prober.go:29] interesting pod/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 04 03:36:38.646950 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:38.646538 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" podUID="d0835f6a-4fcd-48a0-9557-781942c5ddd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 04 03:36:39.650903 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:39.650868 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" Nov 04 03:36:39.651376 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:39.651255 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" Nov 04 03:36:39.652010 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:39.651990 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" Nov 04 03:36:39.824332 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:36:39.824275 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" podStartSLOduration=8.824259772 podStartE2EDuration="8.824259772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:36:33.297110635 +0000 UTC m=+1118.571741399" watchObservedRunningTime="2026-11-04 03:36:39.824259772 +0000 UTC m=+1125.098890534" Nov 04 03:36:46.385675 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:36:46.385634 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:37:01.389725 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:01.389678 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-36ccd3/distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd"] Nov 04 03:37:01.438576 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:01.438536 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-36ccd3/premium-w0-36ccd3-kserve-78d464b6d8-snzsz"] Nov 04 03:37:16.386659 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:37:16.386603 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:37:16.415374 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:16.415336 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-36ccd3/distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd" podUID="d5ba197d-472f-4f3c-bdd1-0b45ae01090f" containerName="main" containerID="cri-o://d46b6b4e84ae4cd9d98d8f2ba89421d423935315e7cb65c595e695b33390fd73" gracePeriod=45 Nov 04 03:37:16.460322 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:16.460285 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-36ccd3/premium-w0-36ccd3-kserve-78d464b6d8-snzsz" podUID="371e82db-7c6c-411d-8f38-988f76bcf821" containerName="main" containerID="cri-o://39d094f1e15705a5b46354883de04c8c4bb49040f78603a3478119bae7af6308" gracePeriod=45 Nov 04 03:37:16.734546 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:16.734522 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-36ccd3/distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd" Nov 04 03:37:16.764844 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:16.764810 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-36ccd3/premium-w0-36ccd3-kserve-78d464b6d8-snzsz" Nov 04 03:37:16.807871 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:16.807836 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-home\" (UniqueName: \"kubernetes.io/empty-dir/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-home\") pod \"d5ba197d-472f-4f3c-bdd1-0b45ae01090f\" (UID: \"d5ba197d-472f-4f3c-bdd1-0b45ae01090f\") " Nov 04 03:37:16.808058 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:16.807974 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-kube-api-access-lg44c\" (UniqueName: \"kubernetes.io/projected/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-kube-api-access-lg44c\") pod \"d5ba197d-472f-4f3c-bdd1-0b45ae01090f\" (UID: \"d5ba197d-472f-4f3c-bdd1-0b45ae01090f\") " Nov 04 03:37:16.808058 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:16.808043 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-model-cache\") pod \"d5ba197d-472f-4f3c-bdd1-0b45ae01090f\" (UID: \"d5ba197d-472f-4f3c-bdd1-0b45ae01090f\") " Nov 04 03:37:16.808179 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:16.808095 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-home" pod "d5ba197d-472f-4f3c-bdd1-0b45ae01090f" (UID: "d5ba197d-472f-4f3c-bdd1-0b45ae01090f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:37:16.808179 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:16.808102 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-tmp-dir\") pod \"d5ba197d-472f-4f3c-bdd1-0b45ae01090f\" (UID: \"d5ba197d-472f-4f3c-bdd1-0b45ae01090f\") " Nov 04 03:37:16.808179 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:16.808152 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-tls-certs\" (UniqueName: \"kubernetes.io/secret/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-tls-certs\") pod \"d5ba197d-472f-4f3c-bdd1-0b45ae01090f\" (UID: \"d5ba197d-472f-4f3c-bdd1-0b45ae01090f\") " Nov 04 03:37:16.808344 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:16.808303 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-dshm\") pod \"d5ba197d-472f-4f3c-bdd1-0b45ae01090f\" (UID: \"d5ba197d-472f-4f3c-bdd1-0b45ae01090f\") " Nov 04 03:37:16.808400 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:16.808340 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-model-cache" pod "d5ba197d-472f-4f3c-bdd1-0b45ae01090f" (UID: "d5ba197d-472f-4f3c-bdd1-0b45ae01090f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:37:16.808400 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:16.808355 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-tmp-dir" pod "d5ba197d-472f-4f3c-bdd1-0b45ae01090f" (UID: "d5ba197d-472f-4f3c-bdd1-0b45ae01090f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:37:16.808848 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:16.808792 5953 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-tmp-dir\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:37:16.808848 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:16.808810 5953 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-home\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:37:16.808848 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:16.808821 5953 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-model-cache\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:37:16.810393 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:16.810370 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-tls-certs" pod "d5ba197d-472f-4f3c-bdd1-0b45ae01090f" (UID: "d5ba197d-472f-4f3c-bdd1-0b45ae01090f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:37:16.810472 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:16.810399 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-dshm" pod "d5ba197d-472f-4f3c-bdd1-0b45ae01090f" (UID: "d5ba197d-472f-4f3c-bdd1-0b45ae01090f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:37:16.810472 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:16.810408 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-kube-api-access-lg44c" pod "d5ba197d-472f-4f3c-bdd1-0b45ae01090f" (UID: "d5ba197d-472f-4f3c-bdd1-0b45ae01090f"). InnerVolumeSpecName "kube-api-access-lg44c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:37:16.910017 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:16.909941 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/371e82db-7c6c-411d-8f38-988f76bcf821-dshm\" (UniqueName: \"kubernetes.io/empty-dir/371e82db-7c6c-411d-8f38-988f76bcf821-dshm\") pod \"371e82db-7c6c-411d-8f38-988f76bcf821\" (UID: \"371e82db-7c6c-411d-8f38-988f76bcf821\") " Nov 04 03:37:16.910017 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:16.909976 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/371e82db-7c6c-411d-8f38-988f76bcf821-home\" (UniqueName: \"kubernetes.io/empty-dir/371e82db-7c6c-411d-8f38-988f76bcf821-home\") pod \"371e82db-7c6c-411d-8f38-988f76bcf821\" (UID: \"371e82db-7c6c-411d-8f38-988f76bcf821\") " Nov 04 03:37:16.910190 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:16.910026 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/371e82db-7c6c-411d-8f38-988f76bcf821-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/371e82db-7c6c-411d-8f38-988f76bcf821-model-cache\") pod \"371e82db-7c6c-411d-8f38-988f76bcf821\" (UID: \"371e82db-7c6c-411d-8f38-988f76bcf821\") " Nov 04 03:37:16.910190 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:16.910071 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/371e82db-7c6c-411d-8f38-988f76bcf821-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/371e82db-7c6c-411d-8f38-988f76bcf821-tmp-dir\") pod \"371e82db-7c6c-411d-8f38-988f76bcf821\" (UID: \"371e82db-7c6c-411d-8f38-988f76bcf821\") " Nov 04 03:37:16.910190 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:16.910160 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/371e82db-7c6c-411d-8f38-988f76bcf821-kube-api-access-rtqtf\" (UniqueName: \"kubernetes.io/projected/371e82db-7c6c-411d-8f38-988f76bcf821-kube-api-access-rtqtf\") pod \"371e82db-7c6c-411d-8f38-988f76bcf821\" (UID: \"371e82db-7c6c-411d-8f38-988f76bcf821\") " Nov 04 03:37:16.910329 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:16.910200 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/371e82db-7c6c-411d-8f38-988f76bcf821-tls-certs\" (UniqueName: \"kubernetes.io/secret/371e82db-7c6c-411d-8f38-988f76bcf821-tls-certs\") pod \"371e82db-7c6c-411d-8f38-988f76bcf821\" (UID: \"371e82db-7c6c-411d-8f38-988f76bcf821\") " Nov 04 03:37:16.910391 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:16.910321 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/371e82db-7c6c-411d-8f38-988f76bcf821-home" pod "371e82db-7c6c-411d-8f38-988f76bcf821" (UID: "371e82db-7c6c-411d-8f38-988f76bcf821"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:37:16.910391 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:16.910334 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/371e82db-7c6c-411d-8f38-988f76bcf821-model-cache" pod "371e82db-7c6c-411d-8f38-988f76bcf821" (UID: "371e82db-7c6c-411d-8f38-988f76bcf821"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:37:16.910499 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:16.910465 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/371e82db-7c6c-411d-8f38-988f76bcf821-tmp-dir" pod "371e82db-7c6c-411d-8f38-988f76bcf821" (UID: "371e82db-7c6c-411d-8f38-988f76bcf821"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:37:16.910764 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:16.910745 5953 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/371e82db-7c6c-411d-8f38-988f76bcf821-home\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:37:16.910805 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:16.910772 5953 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg44c\" (UniqueName: \"kubernetes.io/projected/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-kube-api-access-lg44c\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:37:16.910805 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:16.910785 5953 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/371e82db-7c6c-411d-8f38-988f76bcf821-model-cache\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:37:16.910805 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:16.910798 5953 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/371e82db-7c6c-411d-8f38-988f76bcf821-tmp-dir\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:37:16.910917 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:16.910811 5953 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-tls-certs\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:37:16.910917 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:16.910823 5953 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5ba197d-472f-4f3c-bdd1-0b45ae01090f-dshm\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:37:16.912338 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:16.912310 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/371e82db-7c6c-411d-8f38-988f76bcf821-dshm" pod "371e82db-7c6c-411d-8f38-988f76bcf821" (UID: "371e82db-7c6c-411d-8f38-988f76bcf821"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:37:16.912803 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:16.912767 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/371e82db-7c6c-411d-8f38-988f76bcf821-kube-api-access-rtqtf" pod "371e82db-7c6c-411d-8f38-988f76bcf821" (UID: "371e82db-7c6c-411d-8f38-988f76bcf821"). InnerVolumeSpecName "kube-api-access-rtqtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:37:16.912968 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:16.912943 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/371e82db-7c6c-411d-8f38-988f76bcf821-tls-certs" pod "371e82db-7c6c-411d-8f38-988f76bcf821" (UID: "371e82db-7c6c-411d-8f38-988f76bcf821"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:37:17.011912 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:17.011874 5953 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/371e82db-7c6c-411d-8f38-988f76bcf821-dshm\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:37:17.011912 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:17.011911 5953 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtqtf\" (UniqueName: \"kubernetes.io/projected/371e82db-7c6c-411d-8f38-988f76bcf821-kube-api-access-rtqtf\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:37:17.012114 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:17.011927 5953 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/371e82db-7c6c-411d-8f38-988f76bcf821-tls-certs\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:37:17.506139 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:17.506104 5953 generic.go:356] "Generic (PLEG): container finished" podID="d5ba197d-472f-4f3c-bdd1-0b45ae01090f" containerID="d46b6b4e84ae4cd9d98d8f2ba89421d423935315e7cb65c595e695b33390fd73" exitCode=0 Nov 04 03:37:17.506657 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:17.506196 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-36ccd3/distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd" Nov 04 03:37:17.506657 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:17.506239 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-36ccd3/distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd" event={"ID":"d5ba197d-472f-4f3c-bdd1-0b45ae01090f","Type":"ContainerDied","Data":"d46b6b4e84ae4cd9d98d8f2ba89421d423935315e7cb65c595e695b33390fd73"} Nov 04 03:37:17.506657 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:17.506280 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-36ccd3/distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd" event={"ID":"d5ba197d-472f-4f3c-bdd1-0b45ae01090f","Type":"ContainerDied","Data":"9fb62641e54b4ef0546e59f68d4ce503c5017984337d9e418adc12087af0884c"} Nov 04 03:37:17.506657 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:17.506304 5953 scope.go:122] "RemoveContainer" containerID="d46b6b4e84ae4cd9d98d8f2ba89421d423935315e7cb65c595e695b33390fd73" Nov 04 03:37:17.508871 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:17.508845 5953 generic.go:356] "Generic (PLEG): container finished" podID="371e82db-7c6c-411d-8f38-988f76bcf821" containerID="39d094f1e15705a5b46354883de04c8c4bb49040f78603a3478119bae7af6308" exitCode=0 Nov 04 03:37:17.509170 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:17.509117 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-36ccd3/premium-w0-36ccd3-kserve-78d464b6d8-snzsz" event={"ID":"371e82db-7c6c-411d-8f38-988f76bcf821","Type":"ContainerDied","Data":"39d094f1e15705a5b46354883de04c8c4bb49040f78603a3478119bae7af6308"} Nov 04 03:37:17.509298 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:17.509177 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-36ccd3/premium-w0-36ccd3-kserve-78d464b6d8-snzsz" event={"ID":"371e82db-7c6c-411d-8f38-988f76bcf821","Type":"ContainerDied","Data":"df6888dbb06445ce3f7b4d13ef41b4b9bdd30551acfa39607e967aec52c88a02"} Nov 04 03:37:17.509298 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:17.509253 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-36ccd3/premium-w0-36ccd3-kserve-78d464b6d8-snzsz" Nov 04 03:37:17.520253 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:17.520230 5953 scope.go:122] "RemoveContainer" containerID="d46b6b4e84ae4cd9d98d8f2ba89421d423935315e7cb65c595e695b33390fd73" Nov 04 03:37:17.520573 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:37:17.520552 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d46b6b4e84ae4cd9d98d8f2ba89421d423935315e7cb65c595e695b33390fd73\": container with ID starting with d46b6b4e84ae4cd9d98d8f2ba89421d423935315e7cb65c595e695b33390fd73 not found: ID does not exist" containerID="d46b6b4e84ae4cd9d98d8f2ba89421d423935315e7cb65c595e695b33390fd73" Nov 04 03:37:17.520659 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:17.520584 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d46b6b4e84ae4cd9d98d8f2ba89421d423935315e7cb65c595e695b33390fd73"} err="failed to get container status \"d46b6b4e84ae4cd9d98d8f2ba89421d423935315e7cb65c595e695b33390fd73\": rpc error: code = NotFound desc = could not find container \"d46b6b4e84ae4cd9d98d8f2ba89421d423935315e7cb65c595e695b33390fd73\": container with ID starting with d46b6b4e84ae4cd9d98d8f2ba89421d423935315e7cb65c595e695b33390fd73 not found: ID does not exist" Nov 04 03:37:17.520659 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:17.520602 5953 scope.go:122] "RemoveContainer" containerID="39d094f1e15705a5b46354883de04c8c4bb49040f78603a3478119bae7af6308" Nov 04 03:37:17.530823 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:17.530794 5953 scope.go:122] "RemoveContainer" containerID="39d094f1e15705a5b46354883de04c8c4bb49040f78603a3478119bae7af6308" Nov 04 03:37:17.531180 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:37:17.531158 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39d094f1e15705a5b46354883de04c8c4bb49040f78603a3478119bae7af6308\": container with ID starting with 39d094f1e15705a5b46354883de04c8c4bb49040f78603a3478119bae7af6308 not found: ID does not exist" containerID="39d094f1e15705a5b46354883de04c8c4bb49040f78603a3478119bae7af6308" Nov 04 03:37:17.531249 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:17.531188 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39d094f1e15705a5b46354883de04c8c4bb49040f78603a3478119bae7af6308"} err="failed to get container status \"39d094f1e15705a5b46354883de04c8c4bb49040f78603a3478119bae7af6308\": rpc error: code = NotFound desc = could not find container \"39d094f1e15705a5b46354883de04c8c4bb49040f78603a3478119bae7af6308\": container with ID starting with 39d094f1e15705a5b46354883de04c8c4bb49040f78603a3478119bae7af6308 not found: ID does not exist" Nov 04 03:37:17.674007 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:17.673973 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-36ccd3/premium-w0-36ccd3-kserve-78d464b6d8-snzsz"] Nov 04 03:37:17.723437 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:17.723349 5953 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-36ccd3/premium-w0-36ccd3-kserve-78d464b6d8-snzsz"] Nov 04 03:37:17.833307 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:17.833261 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-36ccd3/distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd"] Nov 04 03:37:17.836984 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:17.836957 5953 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-36ccd3/distinct-2-w0-36ccd3-kserve-7bb457d4f6-vtkbd"] Nov 04 03:37:18.857407 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:18.857375 5953 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="371e82db-7c6c-411d-8f38-988f76bcf821" path="/var/lib/kubelet/pods/371e82db-7c6c-411d-8f38-988f76bcf821/volumes" Nov 04 03:37:18.857859 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:18.857841 5953 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5ba197d-472f-4f3c-bdd1-0b45ae01090f" path="/var/lib/kubelet/pods/d5ba197d-472f-4f3c-bdd1-0b45ae01090f/volumes" Nov 04 03:37:23.314104 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:23.314069 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt"] Nov 04 03:37:23.314751 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:23.314736 5953 cpu_manager.go:414] "containerMap: removing container" podUID="371e82db-7c6c-411d-8f38-988f76bcf821" containerName="main" Nov 04 03:37:23.314824 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:23.314752 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="371e82db-7c6c-411d-8f38-988f76bcf821" containerName="main" Nov 04 03:37:23.314824 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:23.314763 5953 cpu_manager.go:414] "containerMap: removing container" podUID="d5ba197d-472f-4f3c-bdd1-0b45ae01090f" containerName="main" Nov 04 03:37:23.314824 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:23.314769 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5ba197d-472f-4f3c-bdd1-0b45ae01090f" containerName="main" Nov 04 03:37:23.314919 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:23.314856 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="371e82db-7c6c-411d-8f38-988f76bcf821" containerName="main" Nov 04 03:37:23.314919 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:23.314866 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5ba197d-472f-4f3c-bdd1-0b45ae01090f" containerName="main" Nov 04 03:37:23.318318 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:23.318271 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" Nov 04 03:37:23.424324 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:23.424293 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt"] Nov 04 03:37:23.496248 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:23.496207 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de0a15d7-f04e-49e1-a31b-1f46cfa50333-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-swnrt\" (UID: \"de0a15d7-f04e-49e1-a31b-1f46cfa50333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" Nov 04 03:37:23.496248 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:23.496253 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de0a15d7-f04e-49e1-a31b-1f46cfa50333-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-swnrt\" (UID: \"de0a15d7-f04e-49e1-a31b-1f46cfa50333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" Nov 04 03:37:23.496464 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:23.496277 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0a15d7-f04e-49e1-a31b-1f46cfa50333-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-swnrt\" (UID: \"de0a15d7-f04e-49e1-a31b-1f46cfa50333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" Nov 04 03:37:23.496464 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:23.496439 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de0a15d7-f04e-49e1-a31b-1f46cfa50333-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-swnrt\" (UID: \"de0a15d7-f04e-49e1-a31b-1f46cfa50333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" Nov 04 03:37:23.496534 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:23.496472 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de0a15d7-f04e-49e1-a31b-1f46cfa50333-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-swnrt\" (UID: \"de0a15d7-f04e-49e1-a31b-1f46cfa50333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" Nov 04 03:37:23.496568 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:23.496552 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de0a15d7-f04e-49e1-a31b-1f46cfa50333-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-swnrt\" (UID: \"de0a15d7-f04e-49e1-a31b-1f46cfa50333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" Nov 04 03:37:23.496614 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:23.496602 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvzkd\" (UniqueName: \"kubernetes.io/projected/de0a15d7-f04e-49e1-a31b-1f46cfa50333-kube-api-access-cvzkd\") pod \"maas-default-gateway-openshift-default-6dcf757899-swnrt\" (UID: \"de0a15d7-f04e-49e1-a31b-1f46cfa50333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" Nov 04 03:37:23.496691 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:23.496653 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de0a15d7-f04e-49e1-a31b-1f46cfa50333-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-swnrt\" (UID: \"de0a15d7-f04e-49e1-a31b-1f46cfa50333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" Nov 04 03:37:23.496811 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:23.496792 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0a15d7-f04e-49e1-a31b-1f46cfa50333-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-swnrt\" (UID: \"de0a15d7-f04e-49e1-a31b-1f46cfa50333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" Nov 04 03:37:23.597884 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:23.597786 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de0a15d7-f04e-49e1-a31b-1f46cfa50333-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-swnrt\" (UID: \"de0a15d7-f04e-49e1-a31b-1f46cfa50333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" Nov 04 03:37:23.597884 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:23.597879 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0a15d7-f04e-49e1-a31b-1f46cfa50333-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-swnrt\" (UID: \"de0a15d7-f04e-49e1-a31b-1f46cfa50333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" Nov 04 03:37:23.598139 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:23.597910 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de0a15d7-f04e-49e1-a31b-1f46cfa50333-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-swnrt\" (UID: \"de0a15d7-f04e-49e1-a31b-1f46cfa50333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" Nov 04 03:37:23.598139 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:23.597931 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de0a15d7-f04e-49e1-a31b-1f46cfa50333-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-swnrt\" (UID: \"de0a15d7-f04e-49e1-a31b-1f46cfa50333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" Nov 04 03:37:23.598139 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:23.597953 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0a15d7-f04e-49e1-a31b-1f46cfa50333-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-swnrt\" (UID: \"de0a15d7-f04e-49e1-a31b-1f46cfa50333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" Nov 04 03:37:23.598139 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:23.597998 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de0a15d7-f04e-49e1-a31b-1f46cfa50333-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-swnrt\" (UID: \"de0a15d7-f04e-49e1-a31b-1f46cfa50333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" Nov 04 03:37:23.598139 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:23.598020 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de0a15d7-f04e-49e1-a31b-1f46cfa50333-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-swnrt\" (UID: \"de0a15d7-f04e-49e1-a31b-1f46cfa50333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" Nov 04 03:37:23.598139 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:23.598135 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de0a15d7-f04e-49e1-a31b-1f46cfa50333-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-swnrt\" (UID: \"de0a15d7-f04e-49e1-a31b-1f46cfa50333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" Nov 04 03:37:23.598461 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:23.598172 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvzkd\" (UniqueName: \"kubernetes.io/projected/de0a15d7-f04e-49e1-a31b-1f46cfa50333-kube-api-access-cvzkd\") pod \"maas-default-gateway-openshift-default-6dcf757899-swnrt\" (UID: \"de0a15d7-f04e-49e1-a31b-1f46cfa50333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" Nov 04 03:37:23.598461 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:23.598316 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de0a15d7-f04e-49e1-a31b-1f46cfa50333-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-swnrt\" (UID: \"de0a15d7-f04e-49e1-a31b-1f46cfa50333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" Nov 04 03:37:23.598461 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:23.598355 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0a15d7-f04e-49e1-a31b-1f46cfa50333-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-swnrt\" (UID: \"de0a15d7-f04e-49e1-a31b-1f46cfa50333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" Nov 04 03:37:23.598655 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:23.598513 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de0a15d7-f04e-49e1-a31b-1f46cfa50333-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-swnrt\" (UID: \"de0a15d7-f04e-49e1-a31b-1f46cfa50333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" Nov 04 03:37:23.598734 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:23.598717 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de0a15d7-f04e-49e1-a31b-1f46cfa50333-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-swnrt\" (UID: \"de0a15d7-f04e-49e1-a31b-1f46cfa50333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" Nov 04 03:37:23.598809 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:23.598787 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0a15d7-f04e-49e1-a31b-1f46cfa50333-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-swnrt\" (UID: \"de0a15d7-f04e-49e1-a31b-1f46cfa50333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" Nov 04 03:37:23.601294 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:23.601268 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de0a15d7-f04e-49e1-a31b-1f46cfa50333-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-swnrt\" (UID: \"de0a15d7-f04e-49e1-a31b-1f46cfa50333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" Nov 04 03:37:23.601460 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:23.601444 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de0a15d7-f04e-49e1-a31b-1f46cfa50333-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-swnrt\" (UID: \"de0a15d7-f04e-49e1-a31b-1f46cfa50333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" Nov 04 03:37:23.659720 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:23.659678 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de0a15d7-f04e-49e1-a31b-1f46cfa50333-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-swnrt\" (UID: \"de0a15d7-f04e-49e1-a31b-1f46cfa50333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" Nov 04 03:37:23.660149 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:23.660127 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvzkd\" (UniqueName: \"kubernetes.io/projected/de0a15d7-f04e-49e1-a31b-1f46cfa50333-kube-api-access-cvzkd\") pod \"maas-default-gateway-openshift-default-6dcf757899-swnrt\" (UID: \"de0a15d7-f04e-49e1-a31b-1f46cfa50333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" Nov 04 03:37:23.932773 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:23.932672 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" Nov 04 03:37:24.372195 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:24.372152 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt"] Nov 04 03:37:24.373556 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:37:24.373527 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde0a15d7_f04e_49e1_a31b_1f46cfa50333.slice/crio-b87a1e752402539b61cfd032ff6fbd97161323c9f560136e983f0593ba821aec WatchSource:0}: Error finding container b87a1e752402539b61cfd032ff6fbd97161323c9f560136e983f0593ba821aec: Status 404 returned error can't find the container with id b87a1e752402539b61cfd032ff6fbd97161323c9f560136e983f0593ba821aec Nov 04 03:37:24.376236 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:24.376203 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:37:24.376344 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:24.376281 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:37:24.376344 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:24.376324 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:37:24.565470 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:24.565433 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" event={"ID":"de0a15d7-f04e-49e1-a31b-1f46cfa50333","Type":"ContainerStarted","Data":"06cd2aac6a625a1f0bba8f6eb0a250c5258768e20a9c9fbc8b790d2881dd6d10"} Nov 04 03:37:24.565705 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:24.565494 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" event={"ID":"de0a15d7-f04e-49e1-a31b-1f46cfa50333","Type":"ContainerStarted","Data":"b87a1e752402539b61cfd032ff6fbd97161323c9f560136e983f0593ba821aec"} Nov 04 03:37:24.690929 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:24.690817 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" podStartSLOduration=1.690795883 podStartE2EDuration="1.690795883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:37:24.687152752 +0000 UTC m=+1169.961783513" watchObservedRunningTime="2026-11-04 03:37:24.690795883 +0000 UTC m=+1169.965426646" Nov 04 03:37:24.933134 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:24.933101 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" Nov 04 03:37:25.935650 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:25.935569 5953 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-swnrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 04 03:37:25.936057 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:25.935657 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" podUID="de0a15d7-f04e-49e1-a31b-1f46cfa50333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 04 03:37:26.933698 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:26.933660 5953 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-swnrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 04 03:37:26.933895 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:26.933721 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" podUID="de0a15d7-f04e-49e1-a31b-1f46cfa50333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 04 03:37:27.933425 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:27.933385 5953 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-swnrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 04 03:37:27.933849 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:27.933456 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" podUID="de0a15d7-f04e-49e1-a31b-1f46cfa50333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 04 03:37:28.933901 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:28.933866 5953 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-swnrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 04 03:37:28.934321 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:28.933924 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" podUID="de0a15d7-f04e-49e1-a31b-1f46cfa50333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 04 03:37:29.933658 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:29.933582 5953 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-swnrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 04 03:37:29.933883 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:29.933679 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" podUID="de0a15d7-f04e-49e1-a31b-1f46cfa50333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 04 03:37:30.933610 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:30.933574 5953 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-swnrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 04 03:37:30.933610 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:30.933645 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" podUID="de0a15d7-f04e-49e1-a31b-1f46cfa50333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 04 03:37:31.933528 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:31.933491 5953 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-swnrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 04 03:37:31.933784 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:31.933546 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" podUID="de0a15d7-f04e-49e1-a31b-1f46cfa50333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 04 03:37:33.406907 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:33.406866 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" Nov 04 03:37:33.407562 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:33.407290 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" Nov 04 03:37:33.408106 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:33.408086 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-swnrt" Nov 04 03:37:46.387391 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:37:46.387345 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:37:54.839679 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:54.839643 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9"] Nov 04 03:37:54.844013 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:54.843985 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" Nov 04 03:37:54.951794 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:54.951755 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9"] Nov 04 03:37:55.016453 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:55.016410 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-credential-socket\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9\" (UID: \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" Nov 04 03:37:55.016816 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:55.016791 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-istio-envoy\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9\" (UID: \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" Nov 04 03:37:55.016971 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:55.016955 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-istio-podinfo\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9\" (UID: \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" Nov 04 03:37:55.017218 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:55.017198 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-istio-data\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9\" (UID: \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" Nov 04 03:37:55.017359 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:55.017343 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-workload-socket\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9\" (UID: \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" Nov 04 03:37:55.017496 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:55.017482 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-workload-certs\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9\" (UID: \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" Nov 04 03:37:55.017637 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:55.017604 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z7d7\" (UniqueName: \"kubernetes.io/projected/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-kube-api-access-9z7d7\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9\" (UID: \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" Nov 04 03:37:55.018127 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:55.018103 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-istio-token\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9\" (UID: \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" Nov 04 03:37:55.018282 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:55.018266 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-istiod-ca-cert\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9\" (UID: \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" Nov 04 03:37:55.120250 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:55.120166 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-credential-socket\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9\" (UID: \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" Nov 04 03:37:55.120250 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:55.120212 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-istio-envoy\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9\" (UID: \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" Nov 04 03:37:55.120250 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:55.120230 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-istio-podinfo\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9\" (UID: \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" Nov 04 03:37:55.120546 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:55.120292 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-istio-data\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9\" (UID: \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" Nov 04 03:37:55.120546 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:55.120314 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-workload-socket\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9\" (UID: \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" Nov 04 03:37:55.120546 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:55.120337 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-workload-certs\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9\" (UID: \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" Nov 04 03:37:55.120546 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:55.120366 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z7d7\" (UniqueName: \"kubernetes.io/projected/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-kube-api-access-9z7d7\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9\" (UID: \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" Nov 04 03:37:55.120546 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:55.120441 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-istio-token\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9\" (UID: \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" Nov 04 03:37:55.120546 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:55.120465 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-istiod-ca-cert\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9\" (UID: \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" Nov 04 03:37:55.120881 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:55.120768 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-credential-socket\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9\" (UID: \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" Nov 04 03:37:55.120881 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:55.120825 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-istio-data\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9\" (UID: \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" Nov 04 03:37:55.121216 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:55.121195 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-istiod-ca-cert\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9\" (UID: \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" Nov 04 03:37:55.121326 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:55.121301 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-workload-certs\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9\" (UID: \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" Nov 04 03:37:55.121596 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:55.121572 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-workload-socket\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9\" (UID: \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" Nov 04 03:37:55.122997 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:55.122970 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-istio-envoy\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9\" (UID: \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" Nov 04 03:37:55.123120 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:55.123103 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-istio-podinfo\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9\" (UID: \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" Nov 04 03:37:55.182297 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:55.182260 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-istio-token\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9\" (UID: \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" Nov 04 03:37:55.182545 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:55.182522 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z7d7\" (UniqueName: \"kubernetes.io/projected/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-kube-api-access-9z7d7\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9\" (UID: \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" Nov 04 03:37:55.461427 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:55.461339 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" Nov 04 03:37:55.937646 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:55.936644 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9"] Nov 04 03:37:56.010709 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:56.007723 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:37:56.010709 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:56.007815 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:37:56.010709 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:56.007858 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:37:56.824645 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:56.824338 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" event={"ID":"16be7c50-bfd6-48e1-bd0d-776e9723a3e4","Type":"ContainerStarted","Data":"17a9720e6d9f2d93e2939533b7c5193da99c42b4f9d83ca8f89c5af08613f506"} Nov 04 03:37:56.824645 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:56.824412 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" event={"ID":"16be7c50-bfd6-48e1-bd0d-776e9723a3e4","Type":"ContainerStarted","Data":"e22c59ffafb4c4b3ffc4d12a54217e705f6d3830fa83f9866c287c1629688200"} Nov 04 03:37:57.068645 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:57.068258 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" podStartSLOduration=3.0682362420000002 podStartE2EDuration="3.068236242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:37:57.003158755 +0000 UTC m=+1202.277789518" watchObservedRunningTime="2026-11-04 03:37:57.068236242 +0000 UTC m=+1202.342867006" Nov 04 03:37:57.464646 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:57.464580 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" Nov 04 03:37:57.464879 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:57.464780 5953 patch_prober.go:29] interesting pod/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 04 03:37:57.464879 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:57.464847 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" podUID="16be7c50-bfd6-48e1-bd0d-776e9723a3e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 04 03:37:58.499645 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:58.436501 5953 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27" podUID="40a71a57-c1dd-4119-83b9-22527ca0364f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 03:37:58.499645 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:58.474047 5953 patch_prober.go:29] interesting pod/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 04 03:37:58.499645 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:58.474120 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" podUID="16be7c50-bfd6-48e1-bd0d-776e9723a3e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 04 03:37:58.640328 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:58.640278 5953 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27" podUID="40a71a57-c1dd-4119-83b9-22527ca0364f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 04 03:37:59.463190 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:59.463149 5953 patch_prober.go:29] interesting pod/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 04 03:37:59.463403 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:37:59.463220 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" podUID="16be7c50-bfd6-48e1-bd0d-776e9723a3e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 04 03:38:00.462467 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:00.462433 5953 patch_prober.go:29] interesting pod/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 04 03:38:00.462985 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:00.462511 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" podUID="16be7c50-bfd6-48e1-bd0d-776e9723a3e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 04 03:38:01.462292 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:01.462258 5953 patch_prober.go:29] interesting pod/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 04 03:38:01.462509 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:01.462324 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" podUID="16be7c50-bfd6-48e1-bd0d-776e9723a3e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 04 03:38:02.463211 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:02.463175 5953 patch_prober.go:29] interesting pod/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 04 03:38:02.463682 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:02.463247 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" podUID="16be7c50-bfd6-48e1-bd0d-776e9723a3e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 04 03:38:03.126352 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:03.126310 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz"] Nov 04 03:38:03.126848 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:03.126816 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz" podUID="3f4748f7-aaab-4945-9bb1-6e7ae7c287cc" containerName="maas-api" containerID="cri-o://4d4e527beee7b781319aabb25930447914163b9ddca7a1bcca13ad32a0c43037" gracePeriod=30 Nov 04 03:38:03.462885 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:03.462582 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz" Nov 04 03:38:03.485782 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:03.485746 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" Nov 04 03:38:03.486392 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:03.486352 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" Nov 04 03:38:03.487218 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:03.487197 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" Nov 04 03:38:03.506506 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:03.506467 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f4748f7-aaab-4945-9bb1-6e7ae7c287cc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f4748f7-aaab-4945-9bb1-6e7ae7c287cc-metrics-certs\") pod \"3f4748f7-aaab-4945-9bb1-6e7ae7c287cc\" (UID: \"3f4748f7-aaab-4945-9bb1-6e7ae7c287cc\") " Nov 04 03:38:03.506752 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:03.506595 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f4748f7-aaab-4945-9bb1-6e7ae7c287cc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f4748f7-aaab-4945-9bb1-6e7ae7c287cc-openshift-service-ca\") pod \"3f4748f7-aaab-4945-9bb1-6e7ae7c287cc\" (UID: \"3f4748f7-aaab-4945-9bb1-6e7ae7c287cc\") " Nov 04 03:38:03.506752 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:03.506698 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f4748f7-aaab-4945-9bb1-6e7ae7c287cc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f4748f7-aaab-4945-9bb1-6e7ae7c287cc-maas-api-tls\") pod \"3f4748f7-aaab-4945-9bb1-6e7ae7c287cc\" (UID: \"3f4748f7-aaab-4945-9bb1-6e7ae7c287cc\") " Nov 04 03:38:03.506752 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:03.506747 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f4748f7-aaab-4945-9bb1-6e7ae7c287cc-kube-api-access-grrt6\" (UniqueName: \"kubernetes.io/projected/3f4748f7-aaab-4945-9bb1-6e7ae7c287cc-kube-api-access-grrt6\") pod \"3f4748f7-aaab-4945-9bb1-6e7ae7c287cc\" (UID: \"3f4748f7-aaab-4945-9bb1-6e7ae7c287cc\") " Nov 04 03:38:03.507078 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:03.507043 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f4748f7-aaab-4945-9bb1-6e7ae7c287cc-openshift-service-ca" pod "3f4748f7-aaab-4945-9bb1-6e7ae7c287cc" (UID: "3f4748f7-aaab-4945-9bb1-6e7ae7c287cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:38:03.507886 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:03.507419 5953 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f4748f7-aaab-4945-9bb1-6e7ae7c287cc-openshift-service-ca\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:03.509346 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:03.509305 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f4748f7-aaab-4945-9bb1-6e7ae7c287cc-maas-api-tls" pod "3f4748f7-aaab-4945-9bb1-6e7ae7c287cc" (UID: "3f4748f7-aaab-4945-9bb1-6e7ae7c287cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:38:03.509346 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:03.509316 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f4748f7-aaab-4945-9bb1-6e7ae7c287cc-metrics-certs" pod "3f4748f7-aaab-4945-9bb1-6e7ae7c287cc" (UID: "3f4748f7-aaab-4945-9bb1-6e7ae7c287cc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:38:03.509903 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:03.509882 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f4748f7-aaab-4945-9bb1-6e7ae7c287cc-kube-api-access-grrt6" pod "3f4748f7-aaab-4945-9bb1-6e7ae7c287cc" (UID: "3f4748f7-aaab-4945-9bb1-6e7ae7c287cc"). InnerVolumeSpecName "kube-api-access-grrt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:38:03.608328 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:03.608289 5953 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grrt6\" (UniqueName: \"kubernetes.io/projected/3f4748f7-aaab-4945-9bb1-6e7ae7c287cc-kube-api-access-grrt6\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:03.608328 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:03.608323 5953 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f4748f7-aaab-4945-9bb1-6e7ae7c287cc-metrics-certs\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:03.608328 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:03.608337 5953 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f4748f7-aaab-4945-9bb1-6e7ae7c287cc-maas-api-tls\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:03.961446 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:03.961407 5953 generic.go:356] "Generic (PLEG): container finished" podID="3f4748f7-aaab-4945-9bb1-6e7ae7c287cc" containerID="4d4e527beee7b781319aabb25930447914163b9ddca7a1bcca13ad32a0c43037" exitCode=0 Nov 04 03:38:03.961675 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:03.961465 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz" Nov 04 03:38:03.961675 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:03.961483 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz" event={"ID":"3f4748f7-aaab-4945-9bb1-6e7ae7c287cc","Type":"ContainerDied","Data":"4d4e527beee7b781319aabb25930447914163b9ddca7a1bcca13ad32a0c43037"} Nov 04 03:38:03.961675 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:03.961570 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz" event={"ID":"3f4748f7-aaab-4945-9bb1-6e7ae7c287cc","Type":"ContainerDied","Data":"c21de1c15f54e24c84667d72302f1417db8db6c3bdbaf58ee1b4c6e1b752509c"} Nov 04 03:38:03.961675 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:03.961600 5953 scope.go:122] "RemoveContainer" containerID="4d4e527beee7b781319aabb25930447914163b9ddca7a1bcca13ad32a0c43037" Nov 04 03:38:03.973610 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:03.973587 5953 scope.go:122] "RemoveContainer" containerID="4d4e527beee7b781319aabb25930447914163b9ddca7a1bcca13ad32a0c43037" Nov 04 03:38:03.973947 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:38:03.973919 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d4e527beee7b781319aabb25930447914163b9ddca7a1bcca13ad32a0c43037\": container with ID starting with 4d4e527beee7b781319aabb25930447914163b9ddca7a1bcca13ad32a0c43037 not found: ID does not exist" containerID="4d4e527beee7b781319aabb25930447914163b9ddca7a1bcca13ad32a0c43037" Nov 04 03:38:03.974077 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:03.973961 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d4e527beee7b781319aabb25930447914163b9ddca7a1bcca13ad32a0c43037"} err="failed to get container status \"4d4e527beee7b781319aabb25930447914163b9ddca7a1bcca13ad32a0c43037\": rpc error: code = NotFound desc = could not find container \"4d4e527beee7b781319aabb25930447914163b9ddca7a1bcca13ad32a0c43037\": container with ID starting with 4d4e527beee7b781319aabb25930447914163b9ddca7a1bcca13ad32a0c43037 not found: ID does not exist" Nov 04 03:38:04.155803 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:04.145166 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz"] Nov 04 03:38:04.155803 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:04.153634 5953 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-36ccd3-5d45b9d645-79clz"] Nov 04 03:38:04.857307 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:04.857264 5953 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f4748f7-aaab-4945-9bb1-6e7ae7c287cc" path="/var/lib/kubelet/pods/3f4748f7-aaab-4945-9bb1-6e7ae7c287cc/volumes" Nov 04 03:38:08.954489 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:08.954454 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9"] Nov 04 03:38:08.954953 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:08.954644 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" podUID="16be7c50-bfd6-48e1-bd0d-776e9723a3e4" containerName="istio-proxy" containerID="cri-o://17a9720e6d9f2d93e2939533b7c5193da99c42b4f9d83ca8f89c5af08613f506" gracePeriod=30 Nov 04 03:38:08.956893 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:08.956869 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6"] Nov 04 03:38:08.957070 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:08.957048 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" podUID="58497b8e-768f-4133-8c04-a001d4af6ca5" containerName="istio-proxy" containerID="cri-o://a61332668e6b13be0a1b1de3862cf25521822ddaf6b8ac16fd5d096c5d102a43" gracePeriod=30 Nov 04 03:38:14.244922 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.244855 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" Nov 04 03:38:14.273579 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.273551 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" Nov 04 03:38:14.376435 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.376402 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-istio-token\" (UniqueName: \"kubernetes.io/projected/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-istio-token\") pod \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\" (UID: \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\") " Nov 04 03:38:14.376639 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.376444 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58497b8e-768f-4133-8c04-a001d4af6ca5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58497b8e-768f-4133-8c04-a001d4af6ca5-istio-podinfo\") pod \"58497b8e-768f-4133-8c04-a001d4af6ca5\" (UID: \"58497b8e-768f-4133-8c04-a001d4af6ca5\") " Nov 04 03:38:14.376639 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.376483 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58497b8e-768f-4133-8c04-a001d4af6ca5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58497b8e-768f-4133-8c04-a001d4af6ca5-workload-socket\") pod \"58497b8e-768f-4133-8c04-a001d4af6ca5\" (UID: \"58497b8e-768f-4133-8c04-a001d4af6ca5\") " Nov 04 03:38:14.376639 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.376517 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-istio-envoy\") pod \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\" (UID: \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\") " Nov 04 03:38:14.376639 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.376588 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58497b8e-768f-4133-8c04-a001d4af6ca5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58497b8e-768f-4133-8c04-a001d4af6ca5-credential-socket\") pod \"58497b8e-768f-4133-8c04-a001d4af6ca5\" (UID: \"58497b8e-768f-4133-8c04-a001d4af6ca5\") " Nov 04 03:38:14.376895 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.376694 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-istio-data\") pod \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\" (UID: \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\") " Nov 04 03:38:14.376895 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.376754 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58497b8e-768f-4133-8c04-a001d4af6ca5-istio-token\" (UniqueName: \"kubernetes.io/projected/58497b8e-768f-4133-8c04-a001d4af6ca5-istio-token\") pod \"58497b8e-768f-4133-8c04-a001d4af6ca5\" (UID: \"58497b8e-768f-4133-8c04-a001d4af6ca5\") " Nov 04 03:38:14.376895 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.376787 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58497b8e-768f-4133-8c04-a001d4af6ca5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58497b8e-768f-4133-8c04-a001d4af6ca5-workload-certs\") pod \"58497b8e-768f-4133-8c04-a001d4af6ca5\" (UID: \"58497b8e-768f-4133-8c04-a001d4af6ca5\") " Nov 04 03:38:14.376895 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.376823 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58497b8e-768f-4133-8c04-a001d4af6ca5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58497b8e-768f-4133-8c04-a001d4af6ca5-istio-data\") pod \"58497b8e-768f-4133-8c04-a001d4af6ca5\" (UID: \"58497b8e-768f-4133-8c04-a001d4af6ca5\") " Nov 04 03:38:14.376895 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.376849 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-credential-socket\") pod \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\" (UID: \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\") " Nov 04 03:38:14.376895 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.376879 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-workload-certs\") pod \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\" (UID: \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\") " Nov 04 03:38:14.377191 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.376935 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-istiod-ca-cert\") pod \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\" (UID: \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\") " Nov 04 03:38:14.377191 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.376965 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-istio-podinfo\") pod \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\" (UID: \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\") " Nov 04 03:38:14.377191 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.376993 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-kube-api-access-9z7d7\" (UniqueName: \"kubernetes.io/projected/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-kube-api-access-9z7d7\") pod \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\" (UID: \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\") " Nov 04 03:38:14.377191 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.377067 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58497b8e-768f-4133-8c04-a001d4af6ca5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58497b8e-768f-4133-8c04-a001d4af6ca5-istiod-ca-cert\") pod \"58497b8e-768f-4133-8c04-a001d4af6ca5\" (UID: \"58497b8e-768f-4133-8c04-a001d4af6ca5\") " Nov 04 03:38:14.377191 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.377088 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58497b8e-768f-4133-8c04-a001d4af6ca5-credential-socket" pod "58497b8e-768f-4133-8c04-a001d4af6ca5" (UID: "58497b8e-768f-4133-8c04-a001d4af6ca5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:38:14.377191 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.377122 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58497b8e-768f-4133-8c04-a001d4af6ca5-workload-socket" pod "58497b8e-768f-4133-8c04-a001d4af6ca5" (UID: "58497b8e-768f-4133-8c04-a001d4af6ca5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:38:14.377489 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.377230 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-workload-socket\") pod \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\" (UID: \"16be7c50-bfd6-48e1-bd0d-776e9723a3e4\") " Nov 04 03:38:14.377489 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.377265 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58497b8e-768f-4133-8c04-a001d4af6ca5-kube-api-access-4ktwn\" (UniqueName: \"kubernetes.io/projected/58497b8e-768f-4133-8c04-a001d4af6ca5-kube-api-access-4ktwn\") pod \"58497b8e-768f-4133-8c04-a001d4af6ca5\" (UID: \"58497b8e-768f-4133-8c04-a001d4af6ca5\") " Nov 04 03:38:14.377489 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.377318 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58497b8e-768f-4133-8c04-a001d4af6ca5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58497b8e-768f-4133-8c04-a001d4af6ca5-istio-envoy\") pod \"58497b8e-768f-4133-8c04-a001d4af6ca5\" (UID: \"58497b8e-768f-4133-8c04-a001d4af6ca5\") " Nov 04 03:38:14.377489 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.377347 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-credential-socket" pod "16be7c50-bfd6-48e1-bd0d-776e9723a3e4" (UID: "16be7c50-bfd6-48e1-bd0d-776e9723a3e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:38:14.377730 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.377505 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-istio-data" pod "16be7c50-bfd6-48e1-bd0d-776e9723a3e4" (UID: "16be7c50-bfd6-48e1-bd0d-776e9723a3e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:38:14.378023 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.378000 5953 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58497b8e-768f-4133-8c04-a001d4af6ca5-credential-socket\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:14.378182 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.378166 5953 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-istio-data\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:14.378459 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.378445 5953 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-credential-socket\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:14.378563 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.378549 5953 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58497b8e-768f-4133-8c04-a001d4af6ca5-workload-socket\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:14.378672 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.378205 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-workload-socket" pod "16be7c50-bfd6-48e1-bd0d-776e9723a3e4" (UID: "16be7c50-bfd6-48e1-bd0d-776e9723a3e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:38:14.378769 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.378497 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-workload-certs" pod "16be7c50-bfd6-48e1-bd0d-776e9723a3e4" (UID: "16be7c50-bfd6-48e1-bd0d-776e9723a3e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:38:14.378880 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.378578 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58497b8e-768f-4133-8c04-a001d4af6ca5-workload-certs" pod "58497b8e-768f-4133-8c04-a001d4af6ca5" (UID: "58497b8e-768f-4133-8c04-a001d4af6ca5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:38:14.379002 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.378759 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-istiod-ca-cert" pod "16be7c50-bfd6-48e1-bd0d-776e9723a3e4" (UID: "16be7c50-bfd6-48e1-bd0d-776e9723a3e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:38:14.379436 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.379400 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58497b8e-768f-4133-8c04-a001d4af6ca5-istiod-ca-cert" pod "58497b8e-768f-4133-8c04-a001d4af6ca5" (UID: "58497b8e-768f-4133-8c04-a001d4af6ca5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:38:14.379549 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.379505 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58497b8e-768f-4133-8c04-a001d4af6ca5-istio-data" pod "58497b8e-768f-4133-8c04-a001d4af6ca5" (UID: "58497b8e-768f-4133-8c04-a001d4af6ca5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:38:14.379985 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.379933 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-istio-envoy" pod "16be7c50-bfd6-48e1-bd0d-776e9723a3e4" (UID: "16be7c50-bfd6-48e1-bd0d-776e9723a3e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:38:14.380087 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.380008 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-istio-token" pod "16be7c50-bfd6-48e1-bd0d-776e9723a3e4" (UID: "16be7c50-bfd6-48e1-bd0d-776e9723a3e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:38:14.380540 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.380504 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58497b8e-768f-4133-8c04-a001d4af6ca5-istio-token" pod "58497b8e-768f-4133-8c04-a001d4af6ca5" (UID: "58497b8e-768f-4133-8c04-a001d4af6ca5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:38:14.381923 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.381191 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-kube-api-access-9z7d7" pod "16be7c50-bfd6-48e1-bd0d-776e9723a3e4" (UID: "16be7c50-bfd6-48e1-bd0d-776e9723a3e4"). InnerVolumeSpecName "kube-api-access-9z7d7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:38:14.381923 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.381534 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58497b8e-768f-4133-8c04-a001d4af6ca5-istio-podinfo" pod "58497b8e-768f-4133-8c04-a001d4af6ca5" (UID: "58497b8e-768f-4133-8c04-a001d4af6ca5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:38:14.381923 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.381890 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-istio-podinfo" pod "16be7c50-bfd6-48e1-bd0d-776e9723a3e4" (UID: "16be7c50-bfd6-48e1-bd0d-776e9723a3e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:38:14.382666 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.382607 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58497b8e-768f-4133-8c04-a001d4af6ca5-kube-api-access-4ktwn" pod "58497b8e-768f-4133-8c04-a001d4af6ca5" (UID: "58497b8e-768f-4133-8c04-a001d4af6ca5"). InnerVolumeSpecName "kube-api-access-4ktwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:38:14.383215 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.383190 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58497b8e-768f-4133-8c04-a001d4af6ca5-istio-envoy" pod "58497b8e-768f-4133-8c04-a001d4af6ca5" (UID: "58497b8e-768f-4133-8c04-a001d4af6ca5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:38:14.480243 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.480209 5953 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58497b8e-768f-4133-8c04-a001d4af6ca5-istio-token\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:14.480243 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.480237 5953 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58497b8e-768f-4133-8c04-a001d4af6ca5-workload-certs\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:14.480243 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.480248 5953 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58497b8e-768f-4133-8c04-a001d4af6ca5-istio-data\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:14.480495 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.480257 5953 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-workload-certs\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:14.480495 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.480266 5953 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-istiod-ca-cert\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:14.480495 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.480275 5953 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-istio-podinfo\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:14.480495 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.480284 5953 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z7d7\" (UniqueName: \"kubernetes.io/projected/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-kube-api-access-9z7d7\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:14.480495 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.480295 5953 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58497b8e-768f-4133-8c04-a001d4af6ca5-istiod-ca-cert\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:14.480495 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.480305 5953 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-workload-socket\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:14.480495 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.480313 5953 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ktwn\" (UniqueName: \"kubernetes.io/projected/58497b8e-768f-4133-8c04-a001d4af6ca5-kube-api-access-4ktwn\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:14.480495 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.480322 5953 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58497b8e-768f-4133-8c04-a001d4af6ca5-istio-envoy\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:14.480495 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.480330 5953 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-istio-token\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:14.480495 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.480338 5953 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58497b8e-768f-4133-8c04-a001d4af6ca5-istio-podinfo\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:14.480495 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:14.480346 5953 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16be7c50-bfd6-48e1-bd0d-776e9723a3e4-istio-envoy\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:15.034472 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:15.034436 5953 generic.go:356] "Generic (PLEG): container finished" podID="16be7c50-bfd6-48e1-bd0d-776e9723a3e4" containerID="17a9720e6d9f2d93e2939533b7c5193da99c42b4f9d83ca8f89c5af08613f506" exitCode=0 Nov 04 03:38:15.034684 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:15.034484 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" Nov 04 03:38:15.034684 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:15.034530 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" event={"ID":"16be7c50-bfd6-48e1-bd0d-776e9723a3e4","Type":"ContainerDied","Data":"17a9720e6d9f2d93e2939533b7c5193da99c42b4f9d83ca8f89c5af08613f506"} Nov 04 03:38:15.034684 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:15.034589 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9" event={"ID":"16be7c50-bfd6-48e1-bd0d-776e9723a3e4","Type":"ContainerDied","Data":"e22c59ffafb4c4b3ffc4d12a54217e705f6d3830fa83f9866c287c1629688200"} Nov 04 03:38:15.034684 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:15.034641 5953 scope.go:122] "RemoveContainer" containerID="17a9720e6d9f2d93e2939533b7c5193da99c42b4f9d83ca8f89c5af08613f506" Nov 04 03:38:15.036057 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:15.036037 5953 generic.go:356] "Generic (PLEG): container finished" podID="58497b8e-768f-4133-8c04-a001d4af6ca5" containerID="a61332668e6b13be0a1b1de3862cf25521822ddaf6b8ac16fd5d096c5d102a43" exitCode=0 Nov 04 03:38:15.036175 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:15.036083 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" Nov 04 03:38:15.036175 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:15.036126 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" event={"ID":"58497b8e-768f-4133-8c04-a001d4af6ca5","Type":"ContainerDied","Data":"a61332668e6b13be0a1b1de3862cf25521822ddaf6b8ac16fd5d096c5d102a43"} Nov 04 03:38:15.036175 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:15.036170 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6" event={"ID":"58497b8e-768f-4133-8c04-a001d4af6ca5","Type":"ContainerDied","Data":"f8b882dc4a187be5b623ef524efaf657a04d778f06cd42b4bcb99c9bb21f7cbb"} Nov 04 03:38:15.045770 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:15.045742 5953 scope.go:122] "RemoveContainer" containerID="17a9720e6d9f2d93e2939533b7c5193da99c42b4f9d83ca8f89c5af08613f506" Nov 04 03:38:15.046078 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:38:15.046060 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17a9720e6d9f2d93e2939533b7c5193da99c42b4f9d83ca8f89c5af08613f506\": container with ID starting with 17a9720e6d9f2d93e2939533b7c5193da99c42b4f9d83ca8f89c5af08613f506 not found: ID does not exist" containerID="17a9720e6d9f2d93e2939533b7c5193da99c42b4f9d83ca8f89c5af08613f506" Nov 04 03:38:15.046124 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:15.046092 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17a9720e6d9f2d93e2939533b7c5193da99c42b4f9d83ca8f89c5af08613f506"} err="failed to get container status \"17a9720e6d9f2d93e2939533b7c5193da99c42b4f9d83ca8f89c5af08613f506\": rpc error: code = NotFound desc = could not find container \"17a9720e6d9f2d93e2939533b7c5193da99c42b4f9d83ca8f89c5af08613f506\": container with ID starting with 17a9720e6d9f2d93e2939533b7c5193da99c42b4f9d83ca8f89c5af08613f506 not found: ID does not exist" Nov 04 03:38:15.046124 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:15.046110 5953 scope.go:122] "RemoveContainer" containerID="a61332668e6b13be0a1b1de3862cf25521822ddaf6b8ac16fd5d096c5d102a43" Nov 04 03:38:15.055929 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:15.055904 5953 scope.go:122] "RemoveContainer" containerID="a61332668e6b13be0a1b1de3862cf25521822ddaf6b8ac16fd5d096c5d102a43" Nov 04 03:38:15.056268 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:38:15.056247 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a61332668e6b13be0a1b1de3862cf25521822ddaf6b8ac16fd5d096c5d102a43\": container with ID starting with a61332668e6b13be0a1b1de3862cf25521822ddaf6b8ac16fd5d096c5d102a43 not found: ID does not exist" containerID="a61332668e6b13be0a1b1de3862cf25521822ddaf6b8ac16fd5d096c5d102a43" Nov 04 03:38:15.056315 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:15.056277 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a61332668e6b13be0a1b1de3862cf25521822ddaf6b8ac16fd5d096c5d102a43"} err="failed to get container status \"a61332668e6b13be0a1b1de3862cf25521822ddaf6b8ac16fd5d096c5d102a43\": rpc error: code = NotFound desc = could not find container \"a61332668e6b13be0a1b1de3862cf25521822ddaf6b8ac16fd5d096c5d102a43\": container with ID starting with a61332668e6b13be0a1b1de3862cf25521822ddaf6b8ac16fd5d096c5d102a43 not found: ID does not exist" Nov 04 03:38:15.202449 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:15.202366 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9"] Nov 04 03:38:15.252723 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:15.252692 5953 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-5tjk9"] Nov 04 03:38:15.318766 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:15.318725 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6"] Nov 04 03:38:15.321905 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:15.321874 5953 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-zdjg6"] Nov 04 03:38:16.386153 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:38:16.386117 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:38:16.855971 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:16.855931 5953 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16be7c50-bfd6-48e1-bd0d-776e9723a3e4" path="/var/lib/kubelet/pods/16be7c50-bfd6-48e1-bd0d-776e9723a3e4/volumes" Nov 04 03:38:16.856504 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:38:16.856482 5953 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58497b8e-768f-4133-8c04-a001d4af6ca5" path="/var/lib/kubelet/pods/58497b8e-768f-4133-8c04-a001d4af6ca5/volumes" Nov 04 03:38:46.386810 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:38:46.386768 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:39:16.385918 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:39:16.385836 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:39:46.402582 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:39:46.402535 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:39:57.416017 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:57.415978 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp"] Nov 04 03:39:57.416449 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:57.416187 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp" podUID="264f005e-1262-4db5-915b-4887b1425904" containerName="maas-api" containerID="cri-o://52bbc76a7c3a641e3281565a9bf60af625c6a31b90b2aa25850f0cbd0bcb0859" gracePeriod=30 Nov 04 03:39:57.714418 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:57.714381 5953 generic.go:356] "Generic (PLEG): container finished" podID="264f005e-1262-4db5-915b-4887b1425904" containerID="52bbc76a7c3a641e3281565a9bf60af625c6a31b90b2aa25850f0cbd0bcb0859" exitCode=0 Nov 04 03:39:57.714571 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:57.714448 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp" event={"ID":"264f005e-1262-4db5-915b-4887b1425904","Type":"ContainerDied","Data":"52bbc76a7c3a641e3281565a9bf60af625c6a31b90b2aa25850f0cbd0bcb0859"} Nov 04 03:39:57.714571 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:57.714529 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp" event={"ID":"264f005e-1262-4db5-915b-4887b1425904","Type":"ContainerDied","Data":"076135260a1c40915102e30cad30288d7bce8b72cd735b1582e2e290a81b747c"} Nov 04 03:39:57.714571 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:57.714547 5953 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="076135260a1c40915102e30cad30288d7bce8b72cd735b1582e2e290a81b747c" Nov 04 03:39:57.718533 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:57.718514 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp" Nov 04 03:39:57.798431 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:57.798389 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/264f005e-1262-4db5-915b-4887b1425904-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/264f005e-1262-4db5-915b-4887b1425904-openshift-service-ca\") pod \"264f005e-1262-4db5-915b-4887b1425904\" (UID: \"264f005e-1262-4db5-915b-4887b1425904\") " Nov 04 03:39:57.798656 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:57.798460 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/264f005e-1262-4db5-915b-4887b1425904-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/264f005e-1262-4db5-915b-4887b1425904-maas-api-tls\") pod \"264f005e-1262-4db5-915b-4887b1425904\" (UID: \"264f005e-1262-4db5-915b-4887b1425904\") " Nov 04 03:39:57.798656 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:57.798588 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/264f005e-1262-4db5-915b-4887b1425904-kube-api-access-9rrcf\" (UniqueName: \"kubernetes.io/projected/264f005e-1262-4db5-915b-4887b1425904-kube-api-access-9rrcf\") pod \"264f005e-1262-4db5-915b-4887b1425904\" (UID: \"264f005e-1262-4db5-915b-4887b1425904\") " Nov 04 03:39:57.798869 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:57.798842 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/264f005e-1262-4db5-915b-4887b1425904-openshift-service-ca" pod "264f005e-1262-4db5-915b-4887b1425904" (UID: "264f005e-1262-4db5-915b-4887b1425904"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:39:57.798936 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:57.798870 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/264f005e-1262-4db5-915b-4887b1425904-metrics-certs\" (UniqueName: \"kubernetes.io/secret/264f005e-1262-4db5-915b-4887b1425904-metrics-certs\") pod \"264f005e-1262-4db5-915b-4887b1425904\" (UID: \"264f005e-1262-4db5-915b-4887b1425904\") " Nov 04 03:39:57.799412 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:57.799379 5953 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/264f005e-1262-4db5-915b-4887b1425904-openshift-service-ca\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:39:57.800986 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:57.800955 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/264f005e-1262-4db5-915b-4887b1425904-kube-api-access-9rrcf" pod "264f005e-1262-4db5-915b-4887b1425904" (UID: "264f005e-1262-4db5-915b-4887b1425904"). InnerVolumeSpecName "kube-api-access-9rrcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:39:57.801447 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:57.801423 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/264f005e-1262-4db5-915b-4887b1425904-maas-api-tls" pod "264f005e-1262-4db5-915b-4887b1425904" (UID: "264f005e-1262-4db5-915b-4887b1425904"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:39:57.801527 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:57.801483 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/264f005e-1262-4db5-915b-4887b1425904-metrics-certs" pod "264f005e-1262-4db5-915b-4887b1425904" (UID: "264f005e-1262-4db5-915b-4887b1425904"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:39:57.901428 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:57.901104 5953 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/264f005e-1262-4db5-915b-4887b1425904-maas-api-tls\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:39:57.901428 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:57.901142 5953 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rrcf\" (UniqueName: \"kubernetes.io/projected/264f005e-1262-4db5-915b-4887b1425904-kube-api-access-9rrcf\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:39:57.901428 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:57.901157 5953 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/264f005e-1262-4db5-915b-4887b1425904-metrics-certs\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:39:58.572952 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:58.572914 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976"] Nov 04 03:39:58.573366 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:58.573144 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976" podUID="d3220ff1-68e3-4568-a53a-4a9f6e06a5ad" containerName="maas-api" containerID="cri-o://bd854bed2ecb4436e10137b49a38ac8649af6b5ba7fc1acf8860bc53176abdbf" gracePeriod=30 Nov 04 03:39:58.724403 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:58.724371 5953 generic.go:356] "Generic (PLEG): container finished" podID="d3220ff1-68e3-4568-a53a-4a9f6e06a5ad" containerID="bd854bed2ecb4436e10137b49a38ac8649af6b5ba7fc1acf8860bc53176abdbf" exitCode=0 Nov 04 03:39:58.724562 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:58.724439 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976" event={"ID":"d3220ff1-68e3-4568-a53a-4a9f6e06a5ad","Type":"ContainerDied","Data":"bd854bed2ecb4436e10137b49a38ac8649af6b5ba7fc1acf8860bc53176abdbf"} Nov 04 03:39:58.724605 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:58.724564 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp" Nov 04 03:39:58.908159 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:58.908124 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp"] Nov 04 03:39:58.908766 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:58.908743 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976" Nov 04 03:39:58.910606 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:58.910585 5953 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9dda20-6bd97565d-2klwp"] Nov 04 03:39:58.918434 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:58.918411 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3220ff1-68e3-4568-a53a-4a9f6e06a5ad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3220ff1-68e3-4568-a53a-4a9f6e06a5ad-metrics-certs\") pod \"d3220ff1-68e3-4568-a53a-4a9f6e06a5ad\" (UID: \"d3220ff1-68e3-4568-a53a-4a9f6e06a5ad\") " Nov 04 03:39:58.918579 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:58.918457 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3220ff1-68e3-4568-a53a-4a9f6e06a5ad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3220ff1-68e3-4568-a53a-4a9f6e06a5ad-maas-api-tls\") pod \"d3220ff1-68e3-4568-a53a-4a9f6e06a5ad\" (UID: \"d3220ff1-68e3-4568-a53a-4a9f6e06a5ad\") " Nov 04 03:39:58.918579 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:58.918511 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3220ff1-68e3-4568-a53a-4a9f6e06a5ad-kube-api-access-tjvmq\" (UniqueName: \"kubernetes.io/projected/d3220ff1-68e3-4568-a53a-4a9f6e06a5ad-kube-api-access-tjvmq\") pod \"d3220ff1-68e3-4568-a53a-4a9f6e06a5ad\" (UID: \"d3220ff1-68e3-4568-a53a-4a9f6e06a5ad\") " Nov 04 03:39:58.918717 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:58.918593 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3220ff1-68e3-4568-a53a-4a9f6e06a5ad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3220ff1-68e3-4568-a53a-4a9f6e06a5ad-openshift-service-ca\") pod \"d3220ff1-68e3-4568-a53a-4a9f6e06a5ad\" (UID: \"d3220ff1-68e3-4568-a53a-4a9f6e06a5ad\") " Nov 04 03:39:58.919034 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:58.919008 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3220ff1-68e3-4568-a53a-4a9f6e06a5ad-openshift-service-ca" pod "d3220ff1-68e3-4568-a53a-4a9f6e06a5ad" (UID: "d3220ff1-68e3-4568-a53a-4a9f6e06a5ad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:39:58.920855 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:58.920832 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3220ff1-68e3-4568-a53a-4a9f6e06a5ad-maas-api-tls" pod "d3220ff1-68e3-4568-a53a-4a9f6e06a5ad" (UID: "d3220ff1-68e3-4568-a53a-4a9f6e06a5ad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:39:58.921244 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:58.921226 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3220ff1-68e3-4568-a53a-4a9f6e06a5ad-metrics-certs" pod "d3220ff1-68e3-4568-a53a-4a9f6e06a5ad" (UID: "d3220ff1-68e3-4568-a53a-4a9f6e06a5ad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:39:58.921822 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:58.921802 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3220ff1-68e3-4568-a53a-4a9f6e06a5ad-kube-api-access-tjvmq" pod "d3220ff1-68e3-4568-a53a-4a9f6e06a5ad" (UID: "d3220ff1-68e3-4568-a53a-4a9f6e06a5ad"). InnerVolumeSpecName "kube-api-access-tjvmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:39:59.020230 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:59.020186 5953 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3220ff1-68e3-4568-a53a-4a9f6e06a5ad-metrics-certs\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:39:59.020230 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:59.020225 5953 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3220ff1-68e3-4568-a53a-4a9f6e06a5ad-maas-api-tls\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:39:59.020489 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:59.020240 5953 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjvmq\" (UniqueName: \"kubernetes.io/projected/d3220ff1-68e3-4568-a53a-4a9f6e06a5ad-kube-api-access-tjvmq\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:39:59.020489 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:59.020254 5953 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3220ff1-68e3-4568-a53a-4a9f6e06a5ad-openshift-service-ca\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:39:59.731869 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:59.731837 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976" Nov 04 03:39:59.731869 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:59.731848 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976" event={"ID":"d3220ff1-68e3-4568-a53a-4a9f6e06a5ad","Type":"ContainerDied","Data":"b3f2de689d98c0bf2b0994267896c98649f0b0c1dc8ae09bc4cb68c3967a8bb3"} Nov 04 03:39:59.732378 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:59.731911 5953 scope.go:122] "RemoveContainer" containerID="bd854bed2ecb4436e10137b49a38ac8649af6b5ba7fc1acf8860bc53176abdbf" Nov 04 03:39:59.897956 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:59.897921 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976"] Nov 04 03:39:59.948313 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:39:59.948277 5953 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-141891-7b4c8dc5b4-c9976"] Nov 04 03:40:00.857459 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:00.857426 5953 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="264f005e-1262-4db5-915b-4887b1425904" path="/var/lib/kubelet/pods/264f005e-1262-4db5-915b-4887b1425904/volumes" Nov 04 03:40:00.857898 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:00.857883 5953 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3220ff1-68e3-4568-a53a-4a9f6e06a5ad" path="/var/lib/kubelet/pods/d3220ff1-68e3-4568-a53a-4a9f6e06a5ad/volumes" Nov 04 03:40:02.721173 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:02.721128 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4"] Nov 04 03:40:02.721602 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:02.721445 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" podUID="48e61829-0b7d-40a9-b877-398aa633b44f" containerName="istio-proxy" containerID="cri-o://f26695f45bd5653f31814c306f3ca1242d8e3f2ed21ff63550d59b0dbc33dc0d" gracePeriod=30 Nov 04 03:40:03.910643 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:03.910130 5953 patch_prober.go:29] interesting pod/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:40:03.910643 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:03.910202 5953 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" podUID="48e61829-0b7d-40a9-b877-398aa633b44f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:40:04.637530 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:04.637495 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw"] Nov 04 03:40:04.637759 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:04.637731 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" podUID="587e29e9-ccc7-4343-8dd3-2fd2088f5d6c" containerName="istio-proxy" containerID="cri-o://c4d050c2e7885c80a350a02927b7f93ac2b98e4612d814c71883ee8b56ccbc81" gracePeriod=30 Nov 04 03:40:07.998482 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:07.998409 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" Nov 04 03:40:08.136675 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:08.136600 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48e61829-0b7d-40a9-b877-398aa633b44f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48e61829-0b7d-40a9-b877-398aa633b44f-istio-podinfo\") pod \"48e61829-0b7d-40a9-b877-398aa633b44f\" (UID: \"48e61829-0b7d-40a9-b877-398aa633b44f\") " Nov 04 03:40:08.136857 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:08.136685 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48e61829-0b7d-40a9-b877-398aa633b44f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48e61829-0b7d-40a9-b877-398aa633b44f-istio-envoy\") pod \"48e61829-0b7d-40a9-b877-398aa633b44f\" (UID: \"48e61829-0b7d-40a9-b877-398aa633b44f\") " Nov 04 03:40:08.136857 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:08.136717 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48e61829-0b7d-40a9-b877-398aa633b44f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48e61829-0b7d-40a9-b877-398aa633b44f-workload-certs\") pod \"48e61829-0b7d-40a9-b877-398aa633b44f\" (UID: \"48e61829-0b7d-40a9-b877-398aa633b44f\") " Nov 04 03:40:08.136857 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:08.136779 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48e61829-0b7d-40a9-b877-398aa633b44f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48e61829-0b7d-40a9-b877-398aa633b44f-istiod-ca-cert\") pod \"48e61829-0b7d-40a9-b877-398aa633b44f\" (UID: \"48e61829-0b7d-40a9-b877-398aa633b44f\") " Nov 04 03:40:08.136857 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:08.136803 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48e61829-0b7d-40a9-b877-398aa633b44f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e61829-0b7d-40a9-b877-398aa633b44f-workload-socket\") pod \"48e61829-0b7d-40a9-b877-398aa633b44f\" (UID: \"48e61829-0b7d-40a9-b877-398aa633b44f\") " Nov 04 03:40:08.137041 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:08.136907 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48e61829-0b7d-40a9-b877-398aa633b44f-istio-token\" (UniqueName: \"kubernetes.io/projected/48e61829-0b7d-40a9-b877-398aa633b44f-istio-token\") pod \"48e61829-0b7d-40a9-b877-398aa633b44f\" (UID: \"48e61829-0b7d-40a9-b877-398aa633b44f\") " Nov 04 03:40:08.137041 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:08.136945 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48e61829-0b7d-40a9-b877-398aa633b44f-kube-api-access-9pbsg\" (UniqueName: \"kubernetes.io/projected/48e61829-0b7d-40a9-b877-398aa633b44f-kube-api-access-9pbsg\") pod \"48e61829-0b7d-40a9-b877-398aa633b44f\" (UID: \"48e61829-0b7d-40a9-b877-398aa633b44f\") " Nov 04 03:40:08.137041 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:08.137026 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48e61829-0b7d-40a9-b877-398aa633b44f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48e61829-0b7d-40a9-b877-398aa633b44f-istio-data\") pod \"48e61829-0b7d-40a9-b877-398aa633b44f\" (UID: \"48e61829-0b7d-40a9-b877-398aa633b44f\") " Nov 04 03:40:08.137194 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:08.137085 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48e61829-0b7d-40a9-b877-398aa633b44f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e61829-0b7d-40a9-b877-398aa633b44f-credential-socket\") pod \"48e61829-0b7d-40a9-b877-398aa633b44f\" (UID: \"48e61829-0b7d-40a9-b877-398aa633b44f\") " Nov 04 03:40:08.137194 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:08.137100 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48e61829-0b7d-40a9-b877-398aa633b44f-workload-certs" pod "48e61829-0b7d-40a9-b877-398aa633b44f" (UID: "48e61829-0b7d-40a9-b877-398aa633b44f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:40:08.137308 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:08.137209 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48e61829-0b7d-40a9-b877-398aa633b44f-workload-socket" pod "48e61829-0b7d-40a9-b877-398aa633b44f" (UID: "48e61829-0b7d-40a9-b877-398aa633b44f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:40:08.137432 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:08.137407 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48e61829-0b7d-40a9-b877-398aa633b44f-istio-data" pod "48e61829-0b7d-40a9-b877-398aa633b44f" (UID: "48e61829-0b7d-40a9-b877-398aa633b44f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:40:08.137432 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:08.137418 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48e61829-0b7d-40a9-b877-398aa633b44f-istiod-ca-cert" pod "48e61829-0b7d-40a9-b877-398aa633b44f" (UID: "48e61829-0b7d-40a9-b877-398aa633b44f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:40:08.137654 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:08.137583 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48e61829-0b7d-40a9-b877-398aa633b44f-credential-socket" pod "48e61829-0b7d-40a9-b877-398aa633b44f" (UID: "48e61829-0b7d-40a9-b877-398aa633b44f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:40:08.137750 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:08.137722 5953 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e61829-0b7d-40a9-b877-398aa633b44f-workload-socket\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:08.137750 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:08.137740 5953 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48e61829-0b7d-40a9-b877-398aa633b44f-istio-data\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:08.138055 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:08.137750 5953 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e61829-0b7d-40a9-b877-398aa633b44f-credential-socket\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:08.138055 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:08.137761 5953 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48e61829-0b7d-40a9-b877-398aa633b44f-workload-certs\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:08.138055 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:08.137769 5953 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48e61829-0b7d-40a9-b877-398aa633b44f-istiod-ca-cert\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:08.139295 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:08.139266 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48e61829-0b7d-40a9-b877-398aa633b44f-istio-token" pod "48e61829-0b7d-40a9-b877-398aa633b44f" (UID: "48e61829-0b7d-40a9-b877-398aa633b44f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:40:08.139563 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:08.139535 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48e61829-0b7d-40a9-b877-398aa633b44f-istio-envoy" pod "48e61829-0b7d-40a9-b877-398aa633b44f" (UID: "48e61829-0b7d-40a9-b877-398aa633b44f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:40:08.139563 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:08.139550 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48e61829-0b7d-40a9-b877-398aa633b44f-istio-podinfo" pod "48e61829-0b7d-40a9-b877-398aa633b44f" (UID: "48e61829-0b7d-40a9-b877-398aa633b44f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:40:08.139709 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:08.139557 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48e61829-0b7d-40a9-b877-398aa633b44f-kube-api-access-9pbsg" pod "48e61829-0b7d-40a9-b877-398aa633b44f" (UID: "48e61829-0b7d-40a9-b877-398aa633b44f"). InnerVolumeSpecName "kube-api-access-9pbsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:40:08.238630 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:08.238580 5953 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48e61829-0b7d-40a9-b877-398aa633b44f-istio-podinfo\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:08.238630 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:08.238610 5953 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48e61829-0b7d-40a9-b877-398aa633b44f-istio-envoy\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:08.238841 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:08.238649 5953 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48e61829-0b7d-40a9-b877-398aa633b44f-istio-token\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:08.238841 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:08.238658 5953 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pbsg\" (UniqueName: \"kubernetes.io/projected/48e61829-0b7d-40a9-b877-398aa633b44f-kube-api-access-9pbsg\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:08.802554 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:08.802515 5953 generic.go:356] "Generic (PLEG): container finished" podID="48e61829-0b7d-40a9-b877-398aa633b44f" containerID="f26695f45bd5653f31814c306f3ca1242d8e3f2ed21ff63550d59b0dbc33dc0d" exitCode=0 Nov 04 03:40:08.802782 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:08.802566 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" Nov 04 03:40:08.802782 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:08.802599 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" event={"ID":"48e61829-0b7d-40a9-b877-398aa633b44f","Type":"ContainerDied","Data":"f26695f45bd5653f31814c306f3ca1242d8e3f2ed21ff63550d59b0dbc33dc0d"} Nov 04 03:40:08.802782 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:08.802677 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4" event={"ID":"48e61829-0b7d-40a9-b877-398aa633b44f","Type":"ContainerDied","Data":"5d638e5d4dfe057c4583163546e3a4e1f71a2ce97249cf086cf3d9d1d36d423e"} Nov 04 03:40:08.802782 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:08.802703 5953 scope.go:122] "RemoveContainer" containerID="f26695f45bd5653f31814c306f3ca1242d8e3f2ed21ff63550d59b0dbc33dc0d" Nov 04 03:40:08.813364 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:08.813341 5953 scope.go:122] "RemoveContainer" containerID="39fdaf600df1f46a9de017426ecd1afdb4ccbbcc89b7d94bd1dc75887e08c9e8" Nov 04 03:40:08.823740 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:08.823716 5953 scope.go:122] "RemoveContainer" containerID="f26695f45bd5653f31814c306f3ca1242d8e3f2ed21ff63550d59b0dbc33dc0d" Nov 04 03:40:08.824084 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:40:08.824062 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f26695f45bd5653f31814c306f3ca1242d8e3f2ed21ff63550d59b0dbc33dc0d\": container with ID starting with f26695f45bd5653f31814c306f3ca1242d8e3f2ed21ff63550d59b0dbc33dc0d not found: ID does not exist" containerID="f26695f45bd5653f31814c306f3ca1242d8e3f2ed21ff63550d59b0dbc33dc0d" Nov 04 03:40:08.824143 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:08.824096 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f26695f45bd5653f31814c306f3ca1242d8e3f2ed21ff63550d59b0dbc33dc0d"} err="failed to get container status \"f26695f45bd5653f31814c306f3ca1242d8e3f2ed21ff63550d59b0dbc33dc0d\": rpc error: code = NotFound desc = could not find container \"f26695f45bd5653f31814c306f3ca1242d8e3f2ed21ff63550d59b0dbc33dc0d\": container with ID starting with f26695f45bd5653f31814c306f3ca1242d8e3f2ed21ff63550d59b0dbc33dc0d not found: ID does not exist" Nov 04 03:40:08.824143 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:08.824115 5953 scope.go:122] "RemoveContainer" containerID="39fdaf600df1f46a9de017426ecd1afdb4ccbbcc89b7d94bd1dc75887e08c9e8" Nov 04 03:40:08.824395 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:40:08.824375 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39fdaf600df1f46a9de017426ecd1afdb4ccbbcc89b7d94bd1dc75887e08c9e8\": container with ID starting with 39fdaf600df1f46a9de017426ecd1afdb4ccbbcc89b7d94bd1dc75887e08c9e8 not found: ID does not exist" containerID="39fdaf600df1f46a9de017426ecd1afdb4ccbbcc89b7d94bd1dc75887e08c9e8" Nov 04 03:40:08.824455 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:08.824405 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39fdaf600df1f46a9de017426ecd1afdb4ccbbcc89b7d94bd1dc75887e08c9e8"} err="failed to get container status \"39fdaf600df1f46a9de017426ecd1afdb4ccbbcc89b7d94bd1dc75887e08c9e8\": rpc error: code = NotFound desc = could not find container \"39fdaf600df1f46a9de017426ecd1afdb4ccbbcc89b7d94bd1dc75887e08c9e8\": container with ID starting with 39fdaf600df1f46a9de017426ecd1afdb4ccbbcc89b7d94bd1dc75887e08c9e8 not found: ID does not exist" Nov 04 03:40:09.019515 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:09.019478 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4"] Nov 04 03:40:09.069387 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:09.069303 5953 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-141891-openshift-default-ddc68df65-xrds4"] Nov 04 03:40:09.813376 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:09.813338 5953 generic.go:356] "Generic (PLEG): container finished" podID="587e29e9-ccc7-4343-8dd3-2fd2088f5d6c" containerID="c4d050c2e7885c80a350a02927b7f93ac2b98e4612d814c71883ee8b56ccbc81" exitCode=0 Nov 04 03:40:09.813570 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:09.813412 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" event={"ID":"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c","Type":"ContainerDied","Data":"c4d050c2e7885c80a350a02927b7f93ac2b98e4612d814c71883ee8b56ccbc81"} Nov 04 03:40:09.813570 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:09.813486 5953 scope.go:122] "RemoveContainer" containerID="6d0ee94c568256be6c556c2efa13f97288654fdf616fa7ed50a50af976645edb" Nov 04 03:40:09.927754 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:09.927724 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" Nov 04 03:40:10.061726 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:10.061679 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-kube-api-access-2j2z4\" (UniqueName: \"kubernetes.io/projected/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-kube-api-access-2j2z4\") pod \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\" (UID: \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\") " Nov 04 03:40:10.062206 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:10.061749 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-workload-socket\") pod \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\" (UID: \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\") " Nov 04 03:40:10.062206 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:10.061779 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-workload-certs\") pod \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\" (UID: \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\") " Nov 04 03:40:10.062206 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:10.061803 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-credential-socket\") pod \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\" (UID: \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\") " Nov 04 03:40:10.062206 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:10.061822 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-istio-token\" (UniqueName: \"kubernetes.io/projected/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-istio-token\") pod \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\" (UID: \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\") " Nov 04 03:40:10.062206 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:10.061853 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-istio-data\") pod \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\" (UID: \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\") " Nov 04 03:40:10.062206 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:10.061983 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-istiod-ca-cert\") pod \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\" (UID: \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\") " Nov 04 03:40:10.062206 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:10.062182 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-istio-podinfo\") pod \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\" (UID: \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\") " Nov 04 03:40:10.062561 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:10.062161 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-workload-socket" pod "587e29e9-ccc7-4343-8dd3-2fd2088f5d6c" (UID: "587e29e9-ccc7-4343-8dd3-2fd2088f5d6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:40:10.062561 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:10.062212 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-workload-certs" pod "587e29e9-ccc7-4343-8dd3-2fd2088f5d6c" (UID: "587e29e9-ccc7-4343-8dd3-2fd2088f5d6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:40:10.062561 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:10.062233 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-istio-data" pod "587e29e9-ccc7-4343-8dd3-2fd2088f5d6c" (UID: "587e29e9-ccc7-4343-8dd3-2fd2088f5d6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:40:10.062561 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:10.062280 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-istio-envoy\") pod \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\" (UID: \"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c\") " Nov 04 03:40:10.062561 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:10.062302 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-credential-socket" pod "587e29e9-ccc7-4343-8dd3-2fd2088f5d6c" (UID: "587e29e9-ccc7-4343-8dd3-2fd2088f5d6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:40:10.062561 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:10.062508 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-istiod-ca-cert" pod "587e29e9-ccc7-4343-8dd3-2fd2088f5d6c" (UID: "587e29e9-ccc7-4343-8dd3-2fd2088f5d6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:40:10.063063 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:10.063042 5953 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-workload-socket\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:10.063120 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:10.063075 5953 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-workload-certs\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:10.063120 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:10.063094 5953 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-credential-socket\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:10.063120 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:10.063108 5953 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-istio-data\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:10.063227 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:10.063122 5953 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-istiod-ca-cert\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:10.064389 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:10.064368 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-istio-token" pod "587e29e9-ccc7-4343-8dd3-2fd2088f5d6c" (UID: "587e29e9-ccc7-4343-8dd3-2fd2088f5d6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:40:10.064477 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:10.064422 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-istio-envoy" pod "587e29e9-ccc7-4343-8dd3-2fd2088f5d6c" (UID: "587e29e9-ccc7-4343-8dd3-2fd2088f5d6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:40:10.064687 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:10.064665 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-kube-api-access-2j2z4" pod "587e29e9-ccc7-4343-8dd3-2fd2088f5d6c" (UID: "587e29e9-ccc7-4343-8dd3-2fd2088f5d6c"). InnerVolumeSpecName "kube-api-access-2j2z4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:40:10.064782 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:10.064730 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-istio-podinfo" pod "587e29e9-ccc7-4343-8dd3-2fd2088f5d6c" (UID: "587e29e9-ccc7-4343-8dd3-2fd2088f5d6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:40:10.164786 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:10.164696 5953 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-istio-podinfo\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:10.164786 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:10.164727 5953 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-istio-envoy\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:10.164786 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:10.164738 5953 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j2z4\" (UniqueName: \"kubernetes.io/projected/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-kube-api-access-2j2z4\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:10.164786 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:10.164748 5953 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c-istio-token\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:10.821203 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:10.821176 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" Nov 04 03:40:10.821398 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:10.821198 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw" event={"ID":"587e29e9-ccc7-4343-8dd3-2fd2088f5d6c","Type":"ContainerDied","Data":"ccb0f7a5ba6a6bafe42fa99c772e14646e37339318e7998362698d814fabc42e"} Nov 04 03:40:10.821398 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:10.821265 5953 scope.go:122] "RemoveContainer" containerID="c4d050c2e7885c80a350a02927b7f93ac2b98e4612d814c71883ee8b56ccbc81" Nov 04 03:40:10.866137 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:10.866099 5953 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48e61829-0b7d-40a9-b877-398aa633b44f" path="/var/lib/kubelet/pods/48e61829-0b7d-40a9-b877-398aa633b44f/volumes" Nov 04 03:40:10.989399 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:10.989362 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw"] Nov 04 03:40:11.039271 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:11.039234 5953 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-9dda20-openshift-default-84f54f8c7c-rpzmw"] Nov 04 03:40:12.857403 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:12.857369 5953 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="587e29e9-ccc7-4343-8dd3-2fd2088f5d6c" path="/var/lib/kubelet/pods/587e29e9-ccc7-4343-8dd3-2fd2088f5d6c/volumes" Nov 04 03:40:16.386152 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:40:16.386114 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:40:35.049138 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:35.049024 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n"] Nov 04 03:40:35.049753 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:35.049241 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n" podUID="71a3edb7-8071-4977-9977-4adfcceebdcf" containerName="maas-api" containerID="cri-o://b5c3b92e45949e19fa4c37408c1751327ff415e6006d5501c6a1a374a9a1fe2a" gracePeriod=30 Nov 04 03:40:35.382769 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:35.382730 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n" Nov 04 03:40:35.489303 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:35.489263 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71a3edb7-8071-4977-9977-4adfcceebdcf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/71a3edb7-8071-4977-9977-4adfcceebdcf-metrics-certs\") pod \"71a3edb7-8071-4977-9977-4adfcceebdcf\" (UID: \"71a3edb7-8071-4977-9977-4adfcceebdcf\") " Nov 04 03:40:35.489508 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:35.489360 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71a3edb7-8071-4977-9977-4adfcceebdcf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71a3edb7-8071-4977-9977-4adfcceebdcf-maas-api-tls\") pod \"71a3edb7-8071-4977-9977-4adfcceebdcf\" (UID: \"71a3edb7-8071-4977-9977-4adfcceebdcf\") " Nov 04 03:40:35.489508 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:35.489400 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71a3edb7-8071-4977-9977-4adfcceebdcf-kube-api-access-4jzm6\" (UniqueName: \"kubernetes.io/projected/71a3edb7-8071-4977-9977-4adfcceebdcf-kube-api-access-4jzm6\") pod \"71a3edb7-8071-4977-9977-4adfcceebdcf\" (UID: \"71a3edb7-8071-4977-9977-4adfcceebdcf\") " Nov 04 03:40:35.489770 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:35.489511 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71a3edb7-8071-4977-9977-4adfcceebdcf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71a3edb7-8071-4977-9977-4adfcceebdcf-openshift-service-ca\") pod \"71a3edb7-8071-4977-9977-4adfcceebdcf\" (UID: \"71a3edb7-8071-4977-9977-4adfcceebdcf\") " Nov 04 03:40:35.490359 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:35.490326 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71a3edb7-8071-4977-9977-4adfcceebdcf-openshift-service-ca" pod "71a3edb7-8071-4977-9977-4adfcceebdcf" (UID: "71a3edb7-8071-4977-9977-4adfcceebdcf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:40:35.491765 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:35.491742 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71a3edb7-8071-4977-9977-4adfcceebdcf-metrics-certs" pod "71a3edb7-8071-4977-9977-4adfcceebdcf" (UID: "71a3edb7-8071-4977-9977-4adfcceebdcf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:40:35.492403 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:35.492375 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a3edb7-8071-4977-9977-4adfcceebdcf-kube-api-access-4jzm6" pod "71a3edb7-8071-4977-9977-4adfcceebdcf" (UID: "71a3edb7-8071-4977-9977-4adfcceebdcf"). InnerVolumeSpecName "kube-api-access-4jzm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:40:35.492736 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:35.492697 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71a3edb7-8071-4977-9977-4adfcceebdcf-maas-api-tls" pod "71a3edb7-8071-4977-9977-4adfcceebdcf" (UID: "71a3edb7-8071-4977-9977-4adfcceebdcf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:40:35.591011 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:35.590921 5953 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71a3edb7-8071-4977-9977-4adfcceebdcf-maas-api-tls\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:35.591011 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:35.590998 5953 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jzm6\" (UniqueName: \"kubernetes.io/projected/71a3edb7-8071-4977-9977-4adfcceebdcf-kube-api-access-4jzm6\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:35.591205 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:35.591018 5953 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71a3edb7-8071-4977-9977-4adfcceebdcf-openshift-service-ca\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:35.591205 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:35.591034 5953 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71a3edb7-8071-4977-9977-4adfcceebdcf-metrics-certs\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:35.979945 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:35.979848 5953 generic.go:356] "Generic (PLEG): container finished" podID="71a3edb7-8071-4977-9977-4adfcceebdcf" containerID="b5c3b92e45949e19fa4c37408c1751327ff415e6006d5501c6a1a374a9a1fe2a" exitCode=0 Nov 04 03:40:35.979945 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:35.979915 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n" Nov 04 03:40:35.980187 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:35.979936 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n" event={"ID":"71a3edb7-8071-4977-9977-4adfcceebdcf","Type":"ContainerDied","Data":"b5c3b92e45949e19fa4c37408c1751327ff415e6006d5501c6a1a374a9a1fe2a"} Nov 04 03:40:35.980187 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:35.980002 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n" event={"ID":"71a3edb7-8071-4977-9977-4adfcceebdcf","Type":"ContainerDied","Data":"b851a92b5d84b93a95da164b38fba24e3e26737d7a6b19f052e3ac0427093a9f"} Nov 04 03:40:35.980187 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:35.980031 5953 scope.go:122] "RemoveContainer" containerID="b5c3b92e45949e19fa4c37408c1751327ff415e6006d5501c6a1a374a9a1fe2a" Nov 04 03:40:35.994448 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:35.994419 5953 scope.go:122] "RemoveContainer" containerID="b5c3b92e45949e19fa4c37408c1751327ff415e6006d5501c6a1a374a9a1fe2a" Nov 04 03:40:35.995018 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:40:35.994994 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5c3b92e45949e19fa4c37408c1751327ff415e6006d5501c6a1a374a9a1fe2a\": container with ID starting with b5c3b92e45949e19fa4c37408c1751327ff415e6006d5501c6a1a374a9a1fe2a not found: ID does not exist" containerID="b5c3b92e45949e19fa4c37408c1751327ff415e6006d5501c6a1a374a9a1fe2a" Nov 04 03:40:35.995125 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:35.995030 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5c3b92e45949e19fa4c37408c1751327ff415e6006d5501c6a1a374a9a1fe2a"} err="failed to get container status \"b5c3b92e45949e19fa4c37408c1751327ff415e6006d5501c6a1a374a9a1fe2a\": rpc error: code = NotFound desc = could not find container \"b5c3b92e45949e19fa4c37408c1751327ff415e6006d5501c6a1a374a9a1fe2a\": container with ID starting with b5c3b92e45949e19fa4c37408c1751327ff415e6006d5501c6a1a374a9a1fe2a not found: ID does not exist" Nov 04 03:40:36.158994 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:36.158941 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n"] Nov 04 03:40:36.202912 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:36.202876 5953 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-491fe7-c69ccc649-v578n"] Nov 04 03:40:36.880159 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:36.880124 5953 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71a3edb7-8071-4977-9977-4adfcceebdcf" path="/var/lib/kubelet/pods/71a3edb7-8071-4977-9977-4adfcceebdcf/volumes" Nov 04 03:40:42.043155 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:42.043122 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b"] Nov 04 03:40:42.043598 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:42.043329 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" podUID="d0835f6a-4fcd-48a0-9557-781942c5ddd3" containerName="istio-proxy" containerID="cri-o://79035cff882eee49fa0039a6627787c790732f69144db4d4208bb1e1d00f7850" gracePeriod=30 Nov 04 03:40:46.385988 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:40:46.385948 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:40:47.309467 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:47.309400 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" Nov 04 03:40:47.388183 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:47.388142 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0835f6a-4fcd-48a0-9557-781942c5ddd3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0835f6a-4fcd-48a0-9557-781942c5ddd3-credential-socket\") pod \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\" (UID: \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\") " Nov 04 03:40:47.388802 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:47.388258 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0835f6a-4fcd-48a0-9557-781942c5ddd3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0835f6a-4fcd-48a0-9557-781942c5ddd3-istio-envoy\") pod \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\" (UID: \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\") " Nov 04 03:40:47.388802 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:47.388354 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0835f6a-4fcd-48a0-9557-781942c5ddd3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0835f6a-4fcd-48a0-9557-781942c5ddd3-istio-podinfo\") pod \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\" (UID: \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\") " Nov 04 03:40:47.388802 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:47.388395 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0835f6a-4fcd-48a0-9557-781942c5ddd3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0835f6a-4fcd-48a0-9557-781942c5ddd3-workload-socket\") pod \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\" (UID: \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\") " Nov 04 03:40:47.388802 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:47.388529 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0835f6a-4fcd-48a0-9557-781942c5ddd3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0835f6a-4fcd-48a0-9557-781942c5ddd3-workload-certs\") pod \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\" (UID: \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\") " Nov 04 03:40:47.388802 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:47.388563 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0835f6a-4fcd-48a0-9557-781942c5ddd3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0835f6a-4fcd-48a0-9557-781942c5ddd3-istiod-ca-cert\") pod \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\" (UID: \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\") " Nov 04 03:40:47.388802 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:47.388636 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0835f6a-4fcd-48a0-9557-781942c5ddd3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0835f6a-4fcd-48a0-9557-781942c5ddd3-istio-data\") pod \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\" (UID: \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\") " Nov 04 03:40:47.389142 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:47.388804 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0835f6a-4fcd-48a0-9557-781942c5ddd3-istio-token\" (UniqueName: \"kubernetes.io/projected/d0835f6a-4fcd-48a0-9557-781942c5ddd3-istio-token\") pod \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\" (UID: \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\") " Nov 04 03:40:47.389142 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:47.388843 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0835f6a-4fcd-48a0-9557-781942c5ddd3-kube-api-access-mwlg8\" (UniqueName: \"kubernetes.io/projected/d0835f6a-4fcd-48a0-9557-781942c5ddd3-kube-api-access-mwlg8\") pod \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\" (UID: \"d0835f6a-4fcd-48a0-9557-781942c5ddd3\") " Nov 04 03:40:47.389142 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:47.388936 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0835f6a-4fcd-48a0-9557-781942c5ddd3-workload-certs" pod "d0835f6a-4fcd-48a0-9557-781942c5ddd3" (UID: "d0835f6a-4fcd-48a0-9557-781942c5ddd3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:40:47.389298 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:47.389258 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0835f6a-4fcd-48a0-9557-781942c5ddd3-istio-data" pod "d0835f6a-4fcd-48a0-9557-781942c5ddd3" (UID: "d0835f6a-4fcd-48a0-9557-781942c5ddd3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:40:47.389367 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:47.389344 5953 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0835f6a-4fcd-48a0-9557-781942c5ddd3-workload-certs\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:47.389663 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:47.389552 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0835f6a-4fcd-48a0-9557-781942c5ddd3-workload-socket" pod "d0835f6a-4fcd-48a0-9557-781942c5ddd3" (UID: "d0835f6a-4fcd-48a0-9557-781942c5ddd3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:40:47.389752 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:47.389607 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0835f6a-4fcd-48a0-9557-781942c5ddd3-istiod-ca-cert" pod "d0835f6a-4fcd-48a0-9557-781942c5ddd3" (UID: "d0835f6a-4fcd-48a0-9557-781942c5ddd3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:40:47.389872 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:47.389851 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0835f6a-4fcd-48a0-9557-781942c5ddd3-credential-socket" pod "d0835f6a-4fcd-48a0-9557-781942c5ddd3" (UID: "d0835f6a-4fcd-48a0-9557-781942c5ddd3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:40:47.391837 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:47.391810 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0835f6a-4fcd-48a0-9557-781942c5ddd3-kube-api-access-mwlg8" pod "d0835f6a-4fcd-48a0-9557-781942c5ddd3" (UID: "d0835f6a-4fcd-48a0-9557-781942c5ddd3"). InnerVolumeSpecName "kube-api-access-mwlg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:40:47.391949 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:47.391897 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0835f6a-4fcd-48a0-9557-781942c5ddd3-istio-podinfo" pod "d0835f6a-4fcd-48a0-9557-781942c5ddd3" (UID: "d0835f6a-4fcd-48a0-9557-781942c5ddd3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:40:47.392146 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:47.392121 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0835f6a-4fcd-48a0-9557-781942c5ddd3-istio-token" pod "d0835f6a-4fcd-48a0-9557-781942c5ddd3" (UID: "d0835f6a-4fcd-48a0-9557-781942c5ddd3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:40:47.392287 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:47.392255 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0835f6a-4fcd-48a0-9557-781942c5ddd3-istio-envoy" pod "d0835f6a-4fcd-48a0-9557-781942c5ddd3" (UID: "d0835f6a-4fcd-48a0-9557-781942c5ddd3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:40:47.491096 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:47.491060 5953 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0835f6a-4fcd-48a0-9557-781942c5ddd3-credential-socket\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:47.491096 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:47.491088 5953 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0835f6a-4fcd-48a0-9557-781942c5ddd3-istio-envoy\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:47.491096 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:47.491099 5953 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0835f6a-4fcd-48a0-9557-781942c5ddd3-istio-podinfo\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:47.491440 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:47.491109 5953 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0835f6a-4fcd-48a0-9557-781942c5ddd3-workload-socket\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:47.491440 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:47.491118 5953 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0835f6a-4fcd-48a0-9557-781942c5ddd3-istiod-ca-cert\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:47.491440 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:47.491127 5953 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0835f6a-4fcd-48a0-9557-781942c5ddd3-istio-data\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:47.491440 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:47.491135 5953 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0835f6a-4fcd-48a0-9557-781942c5ddd3-istio-token\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:47.491440 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:47.491144 5953 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwlg8\" (UniqueName: \"kubernetes.io/projected/d0835f6a-4fcd-48a0-9557-781942c5ddd3-kube-api-access-mwlg8\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:48.059491 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:48.059459 5953 generic.go:356] "Generic (PLEG): container finished" podID="d0835f6a-4fcd-48a0-9557-781942c5ddd3" containerID="79035cff882eee49fa0039a6627787c790732f69144db4d4208bb1e1d00f7850" exitCode=0 Nov 04 03:40:48.059732 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:48.059506 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" Nov 04 03:40:48.059732 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:48.059553 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" event={"ID":"d0835f6a-4fcd-48a0-9557-781942c5ddd3","Type":"ContainerDied","Data":"79035cff882eee49fa0039a6627787c790732f69144db4d4208bb1e1d00f7850"} Nov 04 03:40:48.059732 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:48.059607 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b" event={"ID":"d0835f6a-4fcd-48a0-9557-781942c5ddd3","Type":"ContainerDied","Data":"0c26a5535e09f4d9466dd7932428ad960c003a17cb55151cc9431387cea62f5e"} Nov 04 03:40:48.059732 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:48.059664 5953 scope.go:122] "RemoveContainer" containerID="79035cff882eee49fa0039a6627787c790732f69144db4d4208bb1e1d00f7850" Nov 04 03:40:48.070477 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:48.070454 5953 scope.go:122] "RemoveContainer" containerID="79035cff882eee49fa0039a6627787c790732f69144db4d4208bb1e1d00f7850" Nov 04 03:40:48.070806 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:40:48.070786 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79035cff882eee49fa0039a6627787c790732f69144db4d4208bb1e1d00f7850\": container with ID starting with 79035cff882eee49fa0039a6627787c790732f69144db4d4208bb1e1d00f7850 not found: ID does not exist" containerID="79035cff882eee49fa0039a6627787c790732f69144db4d4208bb1e1d00f7850" Nov 04 03:40:48.070882 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:48.070816 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79035cff882eee49fa0039a6627787c790732f69144db4d4208bb1e1d00f7850"} err="failed to get container status \"79035cff882eee49fa0039a6627787c790732f69144db4d4208bb1e1d00f7850\": rpc error: code = NotFound desc = could not find container \"79035cff882eee49fa0039a6627787c790732f69144db4d4208bb1e1d00f7850\": container with ID starting with 79035cff882eee49fa0039a6627787c790732f69144db4d4208bb1e1d00f7850 not found: ID does not exist" Nov 04 03:40:48.278974 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:48.278932 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b"] Nov 04 03:40:48.329212 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:48.329131 5953 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-q7s2b"] Nov 04 03:40:48.865603 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:48.865564 5953 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0835f6a-4fcd-48a0-9557-781942c5ddd3" path="/var/lib/kubelet/pods/d0835f6a-4fcd-48a0-9557-781942c5ddd3/volumes" Nov 04 03:40:51.401112 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:51.401077 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b"] Nov 04 03:40:51.401509 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:51.401284 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b" podUID="f014fed4-d5bf-4ce6-8223-90d1d255334d" containerName="maas-api" containerID="cri-o://5ca2fc5d7b362471e3316cbd1e89c7b98e72d57bb638a405d7a06f0f028e4e6d" gracePeriod=30 Nov 04 03:40:51.680667 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:51.680634 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b" Nov 04 03:40:51.740708 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:51.740677 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f014fed4-d5bf-4ce6-8223-90d1d255334d-kube-api-access-c6lv6\" (UniqueName: \"kubernetes.io/projected/f014fed4-d5bf-4ce6-8223-90d1d255334d-kube-api-access-c6lv6\") pod \"f014fed4-d5bf-4ce6-8223-90d1d255334d\" (UID: \"f014fed4-d5bf-4ce6-8223-90d1d255334d\") " Nov 04 03:40:51.740902 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:51.740740 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f014fed4-d5bf-4ce6-8223-90d1d255334d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f014fed4-d5bf-4ce6-8223-90d1d255334d-maas-api-tls\") pod \"f014fed4-d5bf-4ce6-8223-90d1d255334d\" (UID: \"f014fed4-d5bf-4ce6-8223-90d1d255334d\") " Nov 04 03:40:51.740902 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:51.740811 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f014fed4-d5bf-4ce6-8223-90d1d255334d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f014fed4-d5bf-4ce6-8223-90d1d255334d-metrics-certs\") pod \"f014fed4-d5bf-4ce6-8223-90d1d255334d\" (UID: \"f014fed4-d5bf-4ce6-8223-90d1d255334d\") " Nov 04 03:40:51.740902 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:51.740841 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f014fed4-d5bf-4ce6-8223-90d1d255334d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f014fed4-d5bf-4ce6-8223-90d1d255334d-openshift-service-ca\") pod \"f014fed4-d5bf-4ce6-8223-90d1d255334d\" (UID: \"f014fed4-d5bf-4ce6-8223-90d1d255334d\") " Nov 04 03:40:51.741286 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:51.741259 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f014fed4-d5bf-4ce6-8223-90d1d255334d-openshift-service-ca" pod "f014fed4-d5bf-4ce6-8223-90d1d255334d" (UID: "f014fed4-d5bf-4ce6-8223-90d1d255334d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:40:51.741552 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:51.741406 5953 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f014fed4-d5bf-4ce6-8223-90d1d255334d-openshift-service-ca\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:51.743069 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:51.743032 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f014fed4-d5bf-4ce6-8223-90d1d255334d-metrics-certs" pod "f014fed4-d5bf-4ce6-8223-90d1d255334d" (UID: "f014fed4-d5bf-4ce6-8223-90d1d255334d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:40:51.743169 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:51.743069 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f014fed4-d5bf-4ce6-8223-90d1d255334d-maas-api-tls" pod "f014fed4-d5bf-4ce6-8223-90d1d255334d" (UID: "f014fed4-d5bf-4ce6-8223-90d1d255334d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:40:51.743169 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:51.743140 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f014fed4-d5bf-4ce6-8223-90d1d255334d-kube-api-access-c6lv6" pod "f014fed4-d5bf-4ce6-8223-90d1d255334d" (UID: "f014fed4-d5bf-4ce6-8223-90d1d255334d"). InnerVolumeSpecName "kube-api-access-c6lv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:40:51.842270 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:51.842236 5953 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6lv6\" (UniqueName: \"kubernetes.io/projected/f014fed4-d5bf-4ce6-8223-90d1d255334d-kube-api-access-c6lv6\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:51.842270 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:51.842265 5953 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f014fed4-d5bf-4ce6-8223-90d1d255334d-maas-api-tls\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:51.842270 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:51.842275 5953 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f014fed4-d5bf-4ce6-8223-90d1d255334d-metrics-certs\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:52.087601 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:52.087565 5953 generic.go:356] "Generic (PLEG): container finished" podID="f014fed4-d5bf-4ce6-8223-90d1d255334d" containerID="5ca2fc5d7b362471e3316cbd1e89c7b98e72d57bb638a405d7a06f0f028e4e6d" exitCode=0 Nov 04 03:40:52.087802 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:52.087630 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b" Nov 04 03:40:52.087802 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:52.087650 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b" event={"ID":"f014fed4-d5bf-4ce6-8223-90d1d255334d","Type":"ContainerDied","Data":"5ca2fc5d7b362471e3316cbd1e89c7b98e72d57bb638a405d7a06f0f028e4e6d"} Nov 04 03:40:52.087802 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:52.087710 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b" event={"ID":"f014fed4-d5bf-4ce6-8223-90d1d255334d","Type":"ContainerDied","Data":"c958dfb56091107c25130a26d38c75b3250ceae60e068dd82eef8c6f0fc506a7"} Nov 04 03:40:52.087802 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:52.087734 5953 scope.go:122] "RemoveContainer" containerID="5ca2fc5d7b362471e3316cbd1e89c7b98e72d57bb638a405d7a06f0f028e4e6d" Nov 04 03:40:52.098433 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:52.098408 5953 scope.go:122] "RemoveContainer" containerID="5ca2fc5d7b362471e3316cbd1e89c7b98e72d57bb638a405d7a06f0f028e4e6d" Nov 04 03:40:52.098757 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:40:52.098737 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ca2fc5d7b362471e3316cbd1e89c7b98e72d57bb638a405d7a06f0f028e4e6d\": container with ID starting with 5ca2fc5d7b362471e3316cbd1e89c7b98e72d57bb638a405d7a06f0f028e4e6d not found: ID does not exist" containerID="5ca2fc5d7b362471e3316cbd1e89c7b98e72d57bb638a405d7a06f0f028e4e6d" Nov 04 03:40:52.098819 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:52.098768 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ca2fc5d7b362471e3316cbd1e89c7b98e72d57bb638a405d7a06f0f028e4e6d"} err="failed to get container status \"5ca2fc5d7b362471e3316cbd1e89c7b98e72d57bb638a405d7a06f0f028e4e6d\": rpc error: code = NotFound desc = could not find container \"5ca2fc5d7b362471e3316cbd1e89c7b98e72d57bb638a405d7a06f0f028e4e6d\": container with ID starting with 5ca2fc5d7b362471e3316cbd1e89c7b98e72d57bb638a405d7a06f0f028e4e6d not found: ID does not exist" Nov 04 03:40:52.253901 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:52.253870 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b"] Nov 04 03:40:52.257990 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:52.257958 5953 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d1e2ab-687cfc4fdb-4hp7b"] Nov 04 03:40:52.893060 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:52.891661 5953 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f014fed4-d5bf-4ce6-8223-90d1d255334d" path="/var/lib/kubelet/pods/f014fed4-d5bf-4ce6-8223-90d1d255334d/volumes" Nov 04 03:40:55.125185 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:40:55.125148 5953 scope.go:122] "RemoveContainer" containerID="52bbc76a7c3a641e3281565a9bf60af625c6a31b90b2aa25850f0cbd0bcb0859" Nov 04 03:41:11.490315 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:11.490279 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-10f834-6d5d8b948c-fclqf"] Nov 04 03:41:14.549059 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:14.549017 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8"] Nov 04 03:41:14.549561 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:14.549345 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" podUID="4be9f6d1-586c-493a-aa67-76e09cc64ff9" containerName="istio-proxy" containerID="cri-o://61fc1a76802e443bc2ae45f6efeb3a996f4a116056e663049a266d243cbe35df" gracePeriod=30 Nov 04 03:41:16.385597 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:41:16.385564 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:41:16.491334 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:16.491291 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-10f834-6d5d8b948c-fclqf" podUID="9f801ad2-3210-4369-beac-73f789ae5559" containerName="payload-pre-processing" containerID="cri-o://aa9f9c00738e14caf1f261924132f82fe5042be5cce1820ebc055c041005ad54" gracePeriod=25 Nov 04 03:41:19.819945 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:19.819915 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" Nov 04 03:41:19.941955 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:19.941924 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be9f6d1-586c-493a-aa67-76e09cc64ff9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be9f6d1-586c-493a-aa67-76e09cc64ff9-istio-data\") pod \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\" (UID: \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\") " Nov 04 03:41:19.942117 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:19.942034 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be9f6d1-586c-493a-aa67-76e09cc64ff9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be9f6d1-586c-493a-aa67-76e09cc64ff9-workload-certs\") pod \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\" (UID: \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\") " Nov 04 03:41:19.942117 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:19.942061 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be9f6d1-586c-493a-aa67-76e09cc64ff9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be9f6d1-586c-493a-aa67-76e09cc64ff9-credential-socket\") pod \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\" (UID: \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\") " Nov 04 03:41:19.942117 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:19.942082 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4be9f6d1-586c-493a-aa67-76e09cc64ff9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be9f6d1-586c-493a-aa67-76e09cc64ff9-istiod-ca-cert\") pod \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\" (UID: \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\") " Nov 04 03:41:19.942117 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:19.942100 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4be9f6d1-586c-493a-aa67-76e09cc64ff9-kube-api-access-nm4l6\" (UniqueName: \"kubernetes.io/projected/4be9f6d1-586c-493a-aa67-76e09cc64ff9-kube-api-access-nm4l6\") pod \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\" (UID: \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\") " Nov 04 03:41:19.942349 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:19.942127 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4be9f6d1-586c-493a-aa67-76e09cc64ff9-istio-token\" (UniqueName: \"kubernetes.io/projected/4be9f6d1-586c-493a-aa67-76e09cc64ff9-istio-token\") pod \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\" (UID: \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\") " Nov 04 03:41:19.942349 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:19.942181 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be9f6d1-586c-493a-aa67-76e09cc64ff9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be9f6d1-586c-493a-aa67-76e09cc64ff9-istio-envoy\") pod \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\" (UID: \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\") " Nov 04 03:41:19.942349 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:19.942219 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4be9f6d1-586c-493a-aa67-76e09cc64ff9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be9f6d1-586c-493a-aa67-76e09cc64ff9-istio-podinfo\") pod \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\" (UID: \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\") " Nov 04 03:41:19.942349 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:19.942217 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be9f6d1-586c-493a-aa67-76e09cc64ff9-istio-data" pod "4be9f6d1-586c-493a-aa67-76e09cc64ff9" (UID: "4be9f6d1-586c-493a-aa67-76e09cc64ff9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:19.942349 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:19.942282 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be9f6d1-586c-493a-aa67-76e09cc64ff9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be9f6d1-586c-493a-aa67-76e09cc64ff9-workload-socket\") pod \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\" (UID: \"4be9f6d1-586c-493a-aa67-76e09cc64ff9\") " Nov 04 03:41:19.942349 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:19.942338 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be9f6d1-586c-493a-aa67-76e09cc64ff9-credential-socket" pod "4be9f6d1-586c-493a-aa67-76e09cc64ff9" (UID: "4be9f6d1-586c-493a-aa67-76e09cc64ff9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:19.942671 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:19.942438 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be9f6d1-586c-493a-aa67-76e09cc64ff9-workload-certs" pod "4be9f6d1-586c-493a-aa67-76e09cc64ff9" (UID: "4be9f6d1-586c-493a-aa67-76e09cc64ff9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:19.942671 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:19.942637 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be9f6d1-586c-493a-aa67-76e09cc64ff9-workload-socket" pod "4be9f6d1-586c-493a-aa67-76e09cc64ff9" (UID: "4be9f6d1-586c-493a-aa67-76e09cc64ff9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:19.942671 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:19.942543 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4be9f6d1-586c-493a-aa67-76e09cc64ff9-istiod-ca-cert" pod "4be9f6d1-586c-493a-aa67-76e09cc64ff9" (UID: "4be9f6d1-586c-493a-aa67-76e09cc64ff9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:41:19.942905 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:19.942883 5953 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be9f6d1-586c-493a-aa67-76e09cc64ff9-istio-data\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:19.942972 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:19.942911 5953 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be9f6d1-586c-493a-aa67-76e09cc64ff9-workload-certs\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:19.942972 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:19.942929 5953 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be9f6d1-586c-493a-aa67-76e09cc64ff9-credential-socket\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:19.942972 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:19.942941 5953 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be9f6d1-586c-493a-aa67-76e09cc64ff9-istiod-ca-cert\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:19.942972 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:19.942950 5953 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be9f6d1-586c-493a-aa67-76e09cc64ff9-workload-socket\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:19.944668 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:19.944642 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4be9f6d1-586c-493a-aa67-76e09cc64ff9-istio-token" pod "4be9f6d1-586c-493a-aa67-76e09cc64ff9" (UID: "4be9f6d1-586c-493a-aa67-76e09cc64ff9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:41:19.945019 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:19.944997 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4be9f6d1-586c-493a-aa67-76e09cc64ff9-kube-api-access-nm4l6" pod "4be9f6d1-586c-493a-aa67-76e09cc64ff9" (UID: "4be9f6d1-586c-493a-aa67-76e09cc64ff9"). InnerVolumeSpecName "kube-api-access-nm4l6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:41:19.945149 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:19.945127 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be9f6d1-586c-493a-aa67-76e09cc64ff9-istio-envoy" pod "4be9f6d1-586c-493a-aa67-76e09cc64ff9" (UID: "4be9f6d1-586c-493a-aa67-76e09cc64ff9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:19.945221 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:19.945145 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4be9f6d1-586c-493a-aa67-76e09cc64ff9-istio-podinfo" pod "4be9f6d1-586c-493a-aa67-76e09cc64ff9" (UID: "4be9f6d1-586c-493a-aa67-76e09cc64ff9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:41:20.044609 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:20.044568 5953 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm4l6\" (UniqueName: \"kubernetes.io/projected/4be9f6d1-586c-493a-aa67-76e09cc64ff9-kube-api-access-nm4l6\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:20.044609 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:20.044598 5953 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4be9f6d1-586c-493a-aa67-76e09cc64ff9-istio-token\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:20.044609 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:20.044609 5953 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be9f6d1-586c-493a-aa67-76e09cc64ff9-istio-envoy\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:20.044609 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:20.044632 5953 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be9f6d1-586c-493a-aa67-76e09cc64ff9-istio-podinfo\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:20.269096 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:20.269065 5953 generic.go:356] "Generic (PLEG): container finished" podID="4be9f6d1-586c-493a-aa67-76e09cc64ff9" containerID="61fc1a76802e443bc2ae45f6efeb3a996f4a116056e663049a266d243cbe35df" exitCode=0 Nov 04 03:41:20.269303 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:20.269121 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" Nov 04 03:41:20.269303 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:20.269153 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" event={"ID":"4be9f6d1-586c-493a-aa67-76e09cc64ff9","Type":"ContainerDied","Data":"61fc1a76802e443bc2ae45f6efeb3a996f4a116056e663049a266d243cbe35df"} Nov 04 03:41:20.269303 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:20.269215 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8" event={"ID":"4be9f6d1-586c-493a-aa67-76e09cc64ff9","Type":"ContainerDied","Data":"85def7e19d3d28b3684d26c073343e42b8e89b511278aa5e9c15a1a3fc704426"} Nov 04 03:41:20.269303 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:20.269244 5953 scope.go:122] "RemoveContainer" containerID="61fc1a76802e443bc2ae45f6efeb3a996f4a116056e663049a266d243cbe35df" Nov 04 03:41:20.282496 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:20.282470 5953 scope.go:122] "RemoveContainer" containerID="61fc1a76802e443bc2ae45f6efeb3a996f4a116056e663049a266d243cbe35df" Nov 04 03:41:20.283107 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:41:20.283038 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61fc1a76802e443bc2ae45f6efeb3a996f4a116056e663049a266d243cbe35df\": container with ID starting with 61fc1a76802e443bc2ae45f6efeb3a996f4a116056e663049a266d243cbe35df not found: ID does not exist" containerID="61fc1a76802e443bc2ae45f6efeb3a996f4a116056e663049a266d243cbe35df" Nov 04 03:41:20.283107 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:20.283079 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61fc1a76802e443bc2ae45f6efeb3a996f4a116056e663049a266d243cbe35df"} err="failed to get container status \"61fc1a76802e443bc2ae45f6efeb3a996f4a116056e663049a266d243cbe35df\": rpc error: code = NotFound desc = could not find container \"61fc1a76802e443bc2ae45f6efeb3a996f4a116056e663049a266d243cbe35df\": container with ID starting with 61fc1a76802e443bc2ae45f6efeb3a996f4a116056e663049a266d243cbe35df not found: ID does not exist" Nov 04 03:41:20.300138 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:20.300103 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-90051a-cc8656f5-6n6qz"] Nov 04 03:41:20.435830 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:20.435795 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8"] Nov 04 03:41:20.439935 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:20.439906 5953 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-9s9w8"] Nov 04 03:41:20.856925 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:20.856891 5953 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4be9f6d1-586c-493a-aa67-76e09cc64ff9" path="/var/lib/kubelet/pods/4be9f6d1-586c-493a-aa67-76e09cc64ff9/volumes" Nov 04 03:41:24.064478 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:24.064432 5953 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-10f834-6d5d8b948c-fclqf" podUID="9f801ad2-3210-4369-beac-73f789ae5559" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:41:25.301076 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:25.301013 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-90051a-cc8656f5-6n6qz" podUID="8772665e-c7a2-4823-be72-2b93863039ed" containerName="payload-pre-processing" containerID="cri-o://70f4a7e8d22543b5baa574924b0f9d451d9fd35ca825fb8b4d426214e609cab7" gracePeriod=25 Nov 04 03:41:29.248267 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:29.248232 5953 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-90051a-cc8656f5-6n6qz" podUID="8772665e-c7a2-4823-be72-2b93863039ed" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:41:34.064764 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:34.064719 5953 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-10f834-6d5d8b948c-fclqf" podUID="9f801ad2-3210-4369-beac-73f789ae5559" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:41:36.761249 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:36.761218 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-10f834-6d5d8b948c-fclqf" Nov 04 03:41:36.844041 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:36.844001 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f801ad2-3210-4369-beac-73f789ae5559-kube-api-access-4zddk\" (UniqueName: \"kubernetes.io/projected/9f801ad2-3210-4369-beac-73f789ae5559-kube-api-access-4zddk\") pod \"9f801ad2-3210-4369-beac-73f789ae5559\" (UID: \"9f801ad2-3210-4369-beac-73f789ae5559\") " Nov 04 03:41:36.844260 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:36.844088 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f801ad2-3210-4369-beac-73f789ae5559-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f801ad2-3210-4369-beac-73f789ae5559-plugins-config-volume\") pod \"9f801ad2-3210-4369-beac-73f789ae5559\" (UID: \"9f801ad2-3210-4369-beac-73f789ae5559\") " Nov 04 03:41:36.844457 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:36.844430 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f801ad2-3210-4369-beac-73f789ae5559-plugins-config-volume" pod "9f801ad2-3210-4369-beac-73f789ae5559" (UID: "9f801ad2-3210-4369-beac-73f789ae5559"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:41:36.846317 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:36.846296 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f801ad2-3210-4369-beac-73f789ae5559-kube-api-access-4zddk" pod "9f801ad2-3210-4369-beac-73f789ae5559" (UID: "9f801ad2-3210-4369-beac-73f789ae5559"). InnerVolumeSpecName "kube-api-access-4zddk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:41:36.945394 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:36.945355 5953 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f801ad2-3210-4369-beac-73f789ae5559-plugins-config-volume\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:36.945577 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:36.945548 5953 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zddk\" (UniqueName: \"kubernetes.io/projected/9f801ad2-3210-4369-beac-73f789ae5559-kube-api-access-4zddk\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:37.389235 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:37.389203 5953 generic.go:356] "Generic (PLEG): container finished" podID="9f801ad2-3210-4369-beac-73f789ae5559" containerID="aa9f9c00738e14caf1f261924132f82fe5042be5cce1820ebc055c041005ad54" exitCode=0 Nov 04 03:41:37.389429 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:37.389251 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-10f834-6d5d8b948c-fclqf" Nov 04 03:41:37.389429 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:37.389309 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-10f834-6d5d8b948c-fclqf" event={"ID":"9f801ad2-3210-4369-beac-73f789ae5559","Type":"ContainerDied","Data":"aa9f9c00738e14caf1f261924132f82fe5042be5cce1820ebc055c041005ad54"} Nov 04 03:41:37.389429 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:37.389377 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-10f834-6d5d8b948c-fclqf" event={"ID":"9f801ad2-3210-4369-beac-73f789ae5559","Type":"ContainerDied","Data":"501bce7afb4513f3186502c3fad6edb4fe41d5a528645565b61df966bc9d4a47"} Nov 04 03:41:37.389429 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:37.389399 5953 scope.go:122] "RemoveContainer" containerID="aa9f9c00738e14caf1f261924132f82fe5042be5cce1820ebc055c041005ad54" Nov 04 03:41:37.399038 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:37.399020 5953 scope.go:122] "RemoveContainer" containerID="aa9f9c00738e14caf1f261924132f82fe5042be5cce1820ebc055c041005ad54" Nov 04 03:41:37.399331 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:41:37.399311 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa9f9c00738e14caf1f261924132f82fe5042be5cce1820ebc055c041005ad54\": container with ID starting with aa9f9c00738e14caf1f261924132f82fe5042be5cce1820ebc055c041005ad54 not found: ID does not exist" containerID="aa9f9c00738e14caf1f261924132f82fe5042be5cce1820ebc055c041005ad54" Nov 04 03:41:37.399427 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:37.399341 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa9f9c00738e14caf1f261924132f82fe5042be5cce1820ebc055c041005ad54"} err="failed to get container status \"aa9f9c00738e14caf1f261924132f82fe5042be5cce1820ebc055c041005ad54\": rpc error: code = NotFound desc = could not find container \"aa9f9c00738e14caf1f261924132f82fe5042be5cce1820ebc055c041005ad54\": container with ID starting with aa9f9c00738e14caf1f261924132f82fe5042be5cce1820ebc055c041005ad54 not found: ID does not exist" Nov 04 03:41:37.553916 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:37.553877 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-10f834-6d5d8b948c-fclqf"] Nov 04 03:41:37.604128 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:37.604093 5953 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-10f834-6d5d8b948c-fclqf"] Nov 04 03:41:37.940362 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:37.940326 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm"] Nov 04 03:41:37.940778 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:37.940530 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm" podUID="d72a213b-6993-4966-9527-2bb12da6bc8c" containerName="maas-api" containerID="cri-o://98a21828b9879e2cb87296e1af60dfcb6b8649acc43c59d6712132aa4fdac8df" gracePeriod=30 Nov 04 03:41:38.088587 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:38.088545 5953 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm" podUID="d72a213b-6993-4966-9527-2bb12da6bc8c" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.47:8443/health\": dial tcp 10.134.0.47:8443: connect: connection refused" Nov 04 03:41:38.202016 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:38.201963 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm" Nov 04 03:41:38.262325 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:38.262294 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d72a213b-6993-4966-9527-2bb12da6bc8c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d72a213b-6993-4966-9527-2bb12da6bc8c-metrics-certs\") pod \"d72a213b-6993-4966-9527-2bb12da6bc8c\" (UID: \"d72a213b-6993-4966-9527-2bb12da6bc8c\") " Nov 04 03:41:38.262491 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:38.262369 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d72a213b-6993-4966-9527-2bb12da6bc8c-kube-api-access-cg6g2\" (UniqueName: \"kubernetes.io/projected/d72a213b-6993-4966-9527-2bb12da6bc8c-kube-api-access-cg6g2\") pod \"d72a213b-6993-4966-9527-2bb12da6bc8c\" (UID: \"d72a213b-6993-4966-9527-2bb12da6bc8c\") " Nov 04 03:41:38.262491 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:38.262411 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d72a213b-6993-4966-9527-2bb12da6bc8c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d72a213b-6993-4966-9527-2bb12da6bc8c-maas-api-tls\") pod \"d72a213b-6993-4966-9527-2bb12da6bc8c\" (UID: \"d72a213b-6993-4966-9527-2bb12da6bc8c\") " Nov 04 03:41:38.262584 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:38.262522 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d72a213b-6993-4966-9527-2bb12da6bc8c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d72a213b-6993-4966-9527-2bb12da6bc8c-openshift-service-ca\") pod \"d72a213b-6993-4966-9527-2bb12da6bc8c\" (UID: \"d72a213b-6993-4966-9527-2bb12da6bc8c\") " Nov 04 03:41:38.262916 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:38.262891 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d72a213b-6993-4966-9527-2bb12da6bc8c-openshift-service-ca" pod "d72a213b-6993-4966-9527-2bb12da6bc8c" (UID: "d72a213b-6993-4966-9527-2bb12da6bc8c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:41:38.264430 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:38.264411 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d72a213b-6993-4966-9527-2bb12da6bc8c-metrics-certs" pod "d72a213b-6993-4966-9527-2bb12da6bc8c" (UID: "d72a213b-6993-4966-9527-2bb12da6bc8c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:41:38.265108 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:38.265084 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d72a213b-6993-4966-9527-2bb12da6bc8c-maas-api-tls" pod "d72a213b-6993-4966-9527-2bb12da6bc8c" (UID: "d72a213b-6993-4966-9527-2bb12da6bc8c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:41:38.265193 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:38.265107 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d72a213b-6993-4966-9527-2bb12da6bc8c-kube-api-access-cg6g2" pod "d72a213b-6993-4966-9527-2bb12da6bc8c" (UID: "d72a213b-6993-4966-9527-2bb12da6bc8c"). InnerVolumeSpecName "kube-api-access-cg6g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:41:38.364670 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:38.364605 5953 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg6g2\" (UniqueName: \"kubernetes.io/projected/d72a213b-6993-4966-9527-2bb12da6bc8c-kube-api-access-cg6g2\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:38.364670 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:38.364671 5953 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d72a213b-6993-4966-9527-2bb12da6bc8c-maas-api-tls\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:38.365284 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:38.364689 5953 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d72a213b-6993-4966-9527-2bb12da6bc8c-openshift-service-ca\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:38.365284 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:38.364703 5953 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d72a213b-6993-4966-9527-2bb12da6bc8c-metrics-certs\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:38.400828 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:38.400789 5953 generic.go:356] "Generic (PLEG): container finished" podID="d72a213b-6993-4966-9527-2bb12da6bc8c" containerID="98a21828b9879e2cb87296e1af60dfcb6b8649acc43c59d6712132aa4fdac8df" exitCode=0 Nov 04 03:41:38.401129 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:38.401108 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm" event={"ID":"d72a213b-6993-4966-9527-2bb12da6bc8c","Type":"ContainerDied","Data":"98a21828b9879e2cb87296e1af60dfcb6b8649acc43c59d6712132aa4fdac8df"} Nov 04 03:41:38.401320 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:38.401304 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm" event={"ID":"d72a213b-6993-4966-9527-2bb12da6bc8c","Type":"ContainerDied","Data":"f4b9e330c1a2dd6f32b76cbe961019dd794f18b95c922161e59249cb181994a3"} Nov 04 03:41:38.401444 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:38.401432 5953 scope.go:122] "RemoveContainer" containerID="98a21828b9879e2cb87296e1af60dfcb6b8649acc43c59d6712132aa4fdac8df" Nov 04 03:41:38.401735 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:38.401716 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm" Nov 04 03:41:38.412993 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:38.412971 5953 scope.go:122] "RemoveContainer" containerID="98a21828b9879e2cb87296e1af60dfcb6b8649acc43c59d6712132aa4fdac8df" Nov 04 03:41:38.413307 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:41:38.413287 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98a21828b9879e2cb87296e1af60dfcb6b8649acc43c59d6712132aa4fdac8df\": container with ID starting with 98a21828b9879e2cb87296e1af60dfcb6b8649acc43c59d6712132aa4fdac8df not found: ID does not exist" containerID="98a21828b9879e2cb87296e1af60dfcb6b8649acc43c59d6712132aa4fdac8df" Nov 04 03:41:38.413356 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:38.413316 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98a21828b9879e2cb87296e1af60dfcb6b8649acc43c59d6712132aa4fdac8df"} err="failed to get container status \"98a21828b9879e2cb87296e1af60dfcb6b8649acc43c59d6712132aa4fdac8df\": rpc error: code = NotFound desc = could not find container \"98a21828b9879e2cb87296e1af60dfcb6b8649acc43c59d6712132aa4fdac8df\": container with ID starting with 98a21828b9879e2cb87296e1af60dfcb6b8649acc43c59d6712132aa4fdac8df not found: ID does not exist" Nov 04 03:41:38.567169 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:38.567139 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm"] Nov 04 03:41:38.571115 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:38.571085 5953 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-10f834-58799fc8bc-l9zmm"] Nov 04 03:41:38.855890 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:38.855799 5953 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f801ad2-3210-4369-beac-73f789ae5559" path="/var/lib/kubelet/pods/9f801ad2-3210-4369-beac-73f789ae5559/volumes" Nov 04 03:41:38.856283 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:38.856264 5953 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d72a213b-6993-4966-9527-2bb12da6bc8c" path="/var/lib/kubelet/pods/d72a213b-6993-4966-9527-2bb12da6bc8c/volumes" Nov 04 03:41:39.247974 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:39.247893 5953 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-90051a-cc8656f5-6n6qz" podUID="8772665e-c7a2-4823-be72-2b93863039ed" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:41:44.872756 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.872721 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw"] Nov 04 03:41:44.873232 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873217 5953 cpu_manager.go:414] "containerMap: removing container" podUID="9f801ad2-3210-4369-beac-73f789ae5559" containerName="payload-pre-processing" Nov 04 03:41:44.873278 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873234 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f801ad2-3210-4369-beac-73f789ae5559" containerName="payload-pre-processing" Nov 04 03:41:44.873278 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873247 5953 cpu_manager.go:414] "containerMap: removing container" podUID="71a3edb7-8071-4977-9977-4adfcceebdcf" containerName="maas-api" Nov 04 03:41:44.873278 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873253 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71a3edb7-8071-4977-9977-4adfcceebdcf" containerName="maas-api" Nov 04 03:41:44.873278 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873263 5953 cpu_manager.go:414] "containerMap: removing container" podUID="264f005e-1262-4db5-915b-4887b1425904" containerName="maas-api" Nov 04 03:41:44.873278 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873267 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="264f005e-1262-4db5-915b-4887b1425904" containerName="maas-api" Nov 04 03:41:44.873278 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873277 5953 cpu_manager.go:414] "containerMap: removing container" podUID="48e61829-0b7d-40a9-b877-398aa633b44f" containerName="istio-proxy" Nov 04 03:41:44.873448 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873282 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48e61829-0b7d-40a9-b877-398aa633b44f" containerName="istio-proxy" Nov 04 03:41:44.873448 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873290 5953 cpu_manager.go:414] "containerMap: removing container" podUID="587e29e9-ccc7-4343-8dd3-2fd2088f5d6c" containerName="istio-proxy" Nov 04 03:41:44.873448 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873296 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="587e29e9-ccc7-4343-8dd3-2fd2088f5d6c" containerName="istio-proxy" Nov 04 03:41:44.873448 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873310 5953 cpu_manager.go:414] "containerMap: removing container" podUID="d72a213b-6993-4966-9527-2bb12da6bc8c" containerName="maas-api" Nov 04 03:41:44.873448 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873314 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d72a213b-6993-4966-9527-2bb12da6bc8c" containerName="maas-api" Nov 04 03:41:44.873448 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873326 5953 cpu_manager.go:414] "containerMap: removing container" podUID="d0835f6a-4fcd-48a0-9557-781942c5ddd3" containerName="istio-proxy" Nov 04 03:41:44.873448 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873331 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0835f6a-4fcd-48a0-9557-781942c5ddd3" containerName="istio-proxy" Nov 04 03:41:44.873448 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873342 5953 cpu_manager.go:414] "containerMap: removing container" podUID="3f4748f7-aaab-4945-9bb1-6e7ae7c287cc" containerName="maas-api" Nov 04 03:41:44.873448 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873346 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f4748f7-aaab-4945-9bb1-6e7ae7c287cc" containerName="maas-api" Nov 04 03:41:44.873448 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873354 5953 cpu_manager.go:414] "containerMap: removing container" podUID="58497b8e-768f-4133-8c04-a001d4af6ca5" containerName="istio-proxy" Nov 04 03:41:44.873448 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873358 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58497b8e-768f-4133-8c04-a001d4af6ca5" containerName="istio-proxy" Nov 04 03:41:44.873448 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873365 5953 cpu_manager.go:414] "containerMap: removing container" podUID="48e61829-0b7d-40a9-b877-398aa633b44f" containerName="istio-proxy" Nov 04 03:41:44.873448 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873369 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48e61829-0b7d-40a9-b877-398aa633b44f" containerName="istio-proxy" Nov 04 03:41:44.873448 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873379 5953 cpu_manager.go:414] "containerMap: removing container" podUID="16be7c50-bfd6-48e1-bd0d-776e9723a3e4" containerName="istio-proxy" Nov 04 03:41:44.873448 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873383 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16be7c50-bfd6-48e1-bd0d-776e9723a3e4" containerName="istio-proxy" Nov 04 03:41:44.873448 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873391 5953 cpu_manager.go:414] "containerMap: removing container" podUID="587e29e9-ccc7-4343-8dd3-2fd2088f5d6c" containerName="istio-proxy" Nov 04 03:41:44.873448 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873395 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="587e29e9-ccc7-4343-8dd3-2fd2088f5d6c" containerName="istio-proxy" Nov 04 03:41:44.873448 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873403 5953 cpu_manager.go:414] "containerMap: removing container" podUID="f014fed4-d5bf-4ce6-8223-90d1d255334d" containerName="maas-api" Nov 04 03:41:44.873448 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873407 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f014fed4-d5bf-4ce6-8223-90d1d255334d" containerName="maas-api" Nov 04 03:41:44.873448 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873414 5953 cpu_manager.go:414] "containerMap: removing container" podUID="4be9f6d1-586c-493a-aa67-76e09cc64ff9" containerName="istio-proxy" Nov 04 03:41:44.873448 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873418 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4be9f6d1-586c-493a-aa67-76e09cc64ff9" containerName="istio-proxy" Nov 04 03:41:44.873448 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873428 5953 cpu_manager.go:414] "containerMap: removing container" podUID="d3220ff1-68e3-4568-a53a-4a9f6e06a5ad" containerName="maas-api" Nov 04 03:41:44.873448 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873432 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3220ff1-68e3-4568-a53a-4a9f6e06a5ad" containerName="maas-api" Nov 04 03:41:44.874112 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873497 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="d72a213b-6993-4966-9527-2bb12da6bc8c" containerName="maas-api" Nov 04 03:41:44.874112 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873506 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0835f6a-4fcd-48a0-9557-781942c5ddd3" containerName="istio-proxy" Nov 04 03:41:44.874112 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873514 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f4748f7-aaab-4945-9bb1-6e7ae7c287cc" containerName="maas-api" Nov 04 03:41:44.874112 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873520 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="58497b8e-768f-4133-8c04-a001d4af6ca5" containerName="istio-proxy" Nov 04 03:41:44.874112 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873526 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="48e61829-0b7d-40a9-b877-398aa633b44f" containerName="istio-proxy" Nov 04 03:41:44.874112 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873531 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="16be7c50-bfd6-48e1-bd0d-776e9723a3e4" containerName="istio-proxy" Nov 04 03:41:44.874112 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873538 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="587e29e9-ccc7-4343-8dd3-2fd2088f5d6c" containerName="istio-proxy" Nov 04 03:41:44.874112 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873544 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="f014fed4-d5bf-4ce6-8223-90d1d255334d" containerName="maas-api" Nov 04 03:41:44.874112 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873549 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="4be9f6d1-586c-493a-aa67-76e09cc64ff9" containerName="istio-proxy" Nov 04 03:41:44.874112 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873555 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3220ff1-68e3-4568-a53a-4a9f6e06a5ad" containerName="maas-api" Nov 04 03:41:44.874112 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873561 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f801ad2-3210-4369-beac-73f789ae5559" containerName="payload-pre-processing" Nov 04 03:41:44.874112 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873567 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="71a3edb7-8071-4977-9977-4adfcceebdcf" containerName="maas-api" Nov 04 03:41:44.874112 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.873572 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="264f005e-1262-4db5-915b-4887b1425904" containerName="maas-api" Nov 04 03:41:44.876920 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.876905 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" Nov 04 03:41:44.934459 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.934431 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-af788a-openshift-default-dockercfg-826jt\"" Nov 04 03:41:44.937007 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.936981 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/745d29e4-d827-4dc9-b223-2e1f345518ef-workload-socket\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw\" (UID: \"745d29e4-d827-4dc9-b223-2e1f345518ef\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" Nov 04 03:41:44.937135 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.937023 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/745d29e4-d827-4dc9-b223-2e1f345518ef-istio-data\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw\" (UID: \"745d29e4-d827-4dc9-b223-2e1f345518ef\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" Nov 04 03:41:44.937135 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.937052 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/745d29e4-d827-4dc9-b223-2e1f345518ef-istiod-ca-cert\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw\" (UID: \"745d29e4-d827-4dc9-b223-2e1f345518ef\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" Nov 04 03:41:44.937225 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.937205 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/745d29e4-d827-4dc9-b223-2e1f345518ef-istio-token\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw\" (UID: \"745d29e4-d827-4dc9-b223-2e1f345518ef\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" Nov 04 03:41:44.937262 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.937236 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/745d29e4-d827-4dc9-b223-2e1f345518ef-credential-socket\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw\" (UID: \"745d29e4-d827-4dc9-b223-2e1f345518ef\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" Nov 04 03:41:44.937302 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.937257 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/745d29e4-d827-4dc9-b223-2e1f345518ef-workload-certs\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw\" (UID: \"745d29e4-d827-4dc9-b223-2e1f345518ef\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" Nov 04 03:41:44.937405 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.937377 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kzb5\" (UniqueName: \"kubernetes.io/projected/745d29e4-d827-4dc9-b223-2e1f345518ef-kube-api-access-6kzb5\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw\" (UID: \"745d29e4-d827-4dc9-b223-2e1f345518ef\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" Nov 04 03:41:44.937509 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.937420 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/745d29e4-d827-4dc9-b223-2e1f345518ef-istio-envoy\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw\" (UID: \"745d29e4-d827-4dc9-b223-2e1f345518ef\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" Nov 04 03:41:44.937509 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.937451 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/745d29e4-d827-4dc9-b223-2e1f345518ef-istio-podinfo\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw\" (UID: \"745d29e4-d827-4dc9-b223-2e1f345518ef\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" Nov 04 03:41:44.983823 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:44.983793 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw"] Nov 04 03:41:45.038838 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:45.038807 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/745d29e4-d827-4dc9-b223-2e1f345518ef-istio-token\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw\" (UID: \"745d29e4-d827-4dc9-b223-2e1f345518ef\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" Nov 04 03:41:45.038838 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:45.038839 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/745d29e4-d827-4dc9-b223-2e1f345518ef-credential-socket\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw\" (UID: \"745d29e4-d827-4dc9-b223-2e1f345518ef\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" Nov 04 03:41:45.039062 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:45.038854 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/745d29e4-d827-4dc9-b223-2e1f345518ef-workload-certs\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw\" (UID: \"745d29e4-d827-4dc9-b223-2e1f345518ef\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" Nov 04 03:41:45.039062 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:45.038874 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kzb5\" (UniqueName: \"kubernetes.io/projected/745d29e4-d827-4dc9-b223-2e1f345518ef-kube-api-access-6kzb5\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw\" (UID: \"745d29e4-d827-4dc9-b223-2e1f345518ef\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" Nov 04 03:41:45.039062 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:45.038900 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/745d29e4-d827-4dc9-b223-2e1f345518ef-istio-envoy\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw\" (UID: \"745d29e4-d827-4dc9-b223-2e1f345518ef\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" Nov 04 03:41:45.039062 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:45.038930 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/745d29e4-d827-4dc9-b223-2e1f345518ef-istio-podinfo\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw\" (UID: \"745d29e4-d827-4dc9-b223-2e1f345518ef\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" Nov 04 03:41:45.039062 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:45.038997 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/745d29e4-d827-4dc9-b223-2e1f345518ef-workload-socket\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw\" (UID: \"745d29e4-d827-4dc9-b223-2e1f345518ef\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" Nov 04 03:41:45.039062 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:45.039035 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/745d29e4-d827-4dc9-b223-2e1f345518ef-istio-data\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw\" (UID: \"745d29e4-d827-4dc9-b223-2e1f345518ef\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" Nov 04 03:41:45.039062 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:45.039063 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/745d29e4-d827-4dc9-b223-2e1f345518ef-istiod-ca-cert\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw\" (UID: \"745d29e4-d827-4dc9-b223-2e1f345518ef\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" Nov 04 03:41:45.039420 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:45.039256 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/745d29e4-d827-4dc9-b223-2e1f345518ef-credential-socket\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw\" (UID: \"745d29e4-d827-4dc9-b223-2e1f345518ef\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" Nov 04 03:41:45.039420 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:45.039291 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/745d29e4-d827-4dc9-b223-2e1f345518ef-workload-certs\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw\" (UID: \"745d29e4-d827-4dc9-b223-2e1f345518ef\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" Nov 04 03:41:45.039420 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:45.039381 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/745d29e4-d827-4dc9-b223-2e1f345518ef-workload-socket\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw\" (UID: \"745d29e4-d827-4dc9-b223-2e1f345518ef\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" Nov 04 03:41:45.039584 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:45.039427 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/745d29e4-d827-4dc9-b223-2e1f345518ef-istio-data\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw\" (UID: \"745d29e4-d827-4dc9-b223-2e1f345518ef\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" Nov 04 03:41:45.039757 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:45.039740 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/745d29e4-d827-4dc9-b223-2e1f345518ef-istiod-ca-cert\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw\" (UID: \"745d29e4-d827-4dc9-b223-2e1f345518ef\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" Nov 04 03:41:45.041376 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:45.041357 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/745d29e4-d827-4dc9-b223-2e1f345518ef-istio-envoy\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw\" (UID: \"745d29e4-d827-4dc9-b223-2e1f345518ef\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" Nov 04 03:41:45.041560 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:45.041542 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/745d29e4-d827-4dc9-b223-2e1f345518ef-istio-podinfo\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw\" (UID: \"745d29e4-d827-4dc9-b223-2e1f345518ef\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" Nov 04 03:41:45.101230 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:45.101203 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/745d29e4-d827-4dc9-b223-2e1f345518ef-istio-token\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw\" (UID: \"745d29e4-d827-4dc9-b223-2e1f345518ef\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" Nov 04 03:41:45.101461 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:45.101435 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kzb5\" (UniqueName: \"kubernetes.io/projected/745d29e4-d827-4dc9-b223-2e1f345518ef-kube-api-access-6kzb5\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw\" (UID: \"745d29e4-d827-4dc9-b223-2e1f345518ef\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" Nov 04 03:41:45.190120 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:45.190023 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" Nov 04 03:41:45.444507 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:45.444439 5953 generic.go:356] "Generic (PLEG): container finished" podID="8772665e-c7a2-4823-be72-2b93863039ed" containerID="70f4a7e8d22543b5baa574924b0f9d451d9fd35ca825fb8b4d426214e609cab7" exitCode=0 Nov 04 03:41:45.444507 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:45.444482 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-90051a-cc8656f5-6n6qz" event={"ID":"8772665e-c7a2-4823-be72-2b93863039ed","Type":"ContainerDied","Data":"70f4a7e8d22543b5baa574924b0f9d451d9fd35ca825fb8b4d426214e609cab7"} Nov 04 03:41:45.534104 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:45.534082 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-90051a-cc8656f5-6n6qz" Nov 04 03:41:45.634129 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:45.634102 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw"] Nov 04 03:41:45.635968 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:41:45.635942 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod745d29e4_d827_4dc9_b223_2e1f345518ef.slice/crio-c8912f0d3c2a583a637a05f780801afbfcacee219f6d89274f53d3b0f859b04e WatchSource:0}: Error finding container c8912f0d3c2a583a637a05f780801afbfcacee219f6d89274f53d3b0f859b04e: Status 404 returned error can't find the container with id c8912f0d3c2a583a637a05f780801afbfcacee219f6d89274f53d3b0f859b04e Nov 04 03:41:45.638479 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:45.638449 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:41:45.638569 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:45.638514 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:41:45.638569 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:45.638545 5953 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 04 03:41:45.646920 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:45.646896 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8772665e-c7a2-4823-be72-2b93863039ed-kube-api-access-lck6v\" (UniqueName: \"kubernetes.io/projected/8772665e-c7a2-4823-be72-2b93863039ed-kube-api-access-lck6v\") pod \"8772665e-c7a2-4823-be72-2b93863039ed\" (UID: \"8772665e-c7a2-4823-be72-2b93863039ed\") " Nov 04 03:41:45.647001 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:45.646944 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8772665e-c7a2-4823-be72-2b93863039ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8772665e-c7a2-4823-be72-2b93863039ed-plugins-config-volume\") pod \"8772665e-c7a2-4823-be72-2b93863039ed\" (UID: \"8772665e-c7a2-4823-be72-2b93863039ed\") " Nov 04 03:41:45.647290 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:45.647263 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8772665e-c7a2-4823-be72-2b93863039ed-plugins-config-volume" pod "8772665e-c7a2-4823-be72-2b93863039ed" (UID: "8772665e-c7a2-4823-be72-2b93863039ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:41:45.649181 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:45.649155 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8772665e-c7a2-4823-be72-2b93863039ed-kube-api-access-lck6v" pod "8772665e-c7a2-4823-be72-2b93863039ed" (UID: "8772665e-c7a2-4823-be72-2b93863039ed"). InnerVolumeSpecName "kube-api-access-lck6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:41:45.748158 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:45.748129 5953 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lck6v\" (UniqueName: \"kubernetes.io/projected/8772665e-c7a2-4823-be72-2b93863039ed-kube-api-access-lck6v\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:45.748158 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:45.748156 5953 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8772665e-c7a2-4823-be72-2b93863039ed-plugins-config-volume\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:46.385429 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:41:46.385347 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:41:46.451453 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:46.451420 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-90051a-cc8656f5-6n6qz" Nov 04 03:41:46.451691 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:46.451453 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-90051a-cc8656f5-6n6qz" event={"ID":"8772665e-c7a2-4823-be72-2b93863039ed","Type":"ContainerDied","Data":"b4f99e87c8c9e5d76811e9b184efa172c38b9c2914837ba5fcc1ce3072206295"} Nov 04 03:41:46.451691 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:46.451517 5953 scope.go:122] "RemoveContainer" containerID="70f4a7e8d22543b5baa574924b0f9d451d9fd35ca825fb8b4d426214e609cab7" Nov 04 03:41:46.453361 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:46.453339 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" event={"ID":"745d29e4-d827-4dc9-b223-2e1f345518ef","Type":"ContainerStarted","Data":"97d4ceae7ab18a1b9ac458d23a07d31c4b351f9246767a755e044b609c02177d"} Nov 04 03:41:46.453459 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:46.453379 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" event={"ID":"745d29e4-d827-4dc9-b223-2e1f345518ef","Type":"ContainerStarted","Data":"c8912f0d3c2a583a637a05f780801afbfcacee219f6d89274f53d3b0f859b04e"} Nov 04 03:41:46.654262 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:46.648817 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" podStartSLOduration=2.648796657 podStartE2EDuration="2.648796657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:41:46.647285613 +0000 UTC m=+1431.921916375" watchObservedRunningTime="2026-11-04 03:41:46.648796657 +0000 UTC m=+1431.923427419" Nov 04 03:41:46.761262 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:46.761228 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-90051a-cc8656f5-6n6qz"] Nov 04 03:41:46.764648 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:46.764609 5953 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-90051a-cc8656f5-6n6qz"] Nov 04 03:41:46.855889 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:46.855856 5953 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8772665e-c7a2-4823-be72-2b93863039ed" path="/var/lib/kubelet/pods/8772665e-c7a2-4823-be72-2b93863039ed/volumes" Nov 04 03:41:47.191141 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:47.191115 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" Nov 04 03:41:47.192591 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:47.192569 5953 patch_prober.go:29] interesting pod/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 04 03:41:47.192752 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:47.192614 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" podUID="745d29e4-d827-4dc9-b223-2e1f345518ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 04 03:41:48.191305 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:48.191263 5953 patch_prober.go:29] interesting pod/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 04 03:41:48.191863 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:48.191325 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" podUID="745d29e4-d827-4dc9-b223-2e1f345518ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 04 03:41:49.191011 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:49.190974 5953 patch_prober.go:29] interesting pod/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 04 03:41:49.191196 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:49.191026 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" podUID="745d29e4-d827-4dc9-b223-2e1f345518ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 04 03:41:50.191126 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:50.191092 5953 patch_prober.go:29] interesting pod/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 04 03:41:50.191518 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:50.191156 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" podUID="745d29e4-d827-4dc9-b223-2e1f345518ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 04 03:41:50.550602 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:50.550569 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8"] Nov 04 03:41:50.551308 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:50.551257 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" podUID="e9d98901-eb52-4bd0-a031-640623bd4272" containerName="istio-proxy" containerID="cri-o://631110178fbc6ef1a26586f6ca6109f7d52e464849a0ed7729fe4bf127a1a46b" gracePeriod=30 Nov 04 03:41:51.190556 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:51.190518 5953 patch_prober.go:29] interesting pod/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 04 03:41:51.190916 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:51.190881 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" podUID="745d29e4-d827-4dc9-b223-2e1f345518ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 04 03:41:52.190914 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:52.190874 5953 patch_prober.go:29] interesting pod/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 04 03:41:52.191355 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:52.190946 5953 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" podUID="745d29e4-d827-4dc9-b223-2e1f345518ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 04 03:41:52.801445 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:52.801400 5953 patch_prober.go:29] interesting pod/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:41:52.801699 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:52.801474 5953 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" podUID="e9d98901-eb52-4bd0-a031-640623bd4272" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:41:53.235314 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:53.235216 5953 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" Nov 04 03:41:53.235937 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:53.235908 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" Nov 04 03:41:53.236795 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:53.236762 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" Nov 04 03:41:53.818523 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:53.817022 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw"] Nov 04 03:41:54.349240 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:54.349205 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w"] Nov 04 03:41:54.349684 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:54.349410 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" podUID="e573d41f-40a7-4e92-880d-a0204b14633a" containerName="istio-proxy" containerID="cri-o://e02083754d8c50a5bf60ad89092401202602e4d2e292bf9ad57b909627192e31" gracePeriod=30 Nov 04 03:41:54.520307 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:54.520270 5953 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" secret="" err="secret \"e2e-worker-w2-af788a-openshift-default-dockercfg-826jt\" not found" Nov 04 03:41:55.525383 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:55.525329 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" podUID="745d29e4-d827-4dc9-b223-2e1f345518ef" containerName="istio-proxy" containerID="cri-o://97d4ceae7ab18a1b9ac458d23a07d31c4b351f9246767a755e044b609c02177d" gracePeriod=30 Nov 04 03:41:55.819410 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:55.819387 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" Nov 04 03:41:55.992466 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:55.992428 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9d98901-eb52-4bd0-a031-640623bd4272-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9d98901-eb52-4bd0-a031-640623bd4272-istio-envoy\") pod \"e9d98901-eb52-4bd0-a031-640623bd4272\" (UID: \"e9d98901-eb52-4bd0-a031-640623bd4272\") " Nov 04 03:41:55.992673 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:55.992478 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9d98901-eb52-4bd0-a031-640623bd4272-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9d98901-eb52-4bd0-a031-640623bd4272-istio-podinfo\") pod \"e9d98901-eb52-4bd0-a031-640623bd4272\" (UID: \"e9d98901-eb52-4bd0-a031-640623bd4272\") " Nov 04 03:41:55.992673 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:55.992581 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9d98901-eb52-4bd0-a031-640623bd4272-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9d98901-eb52-4bd0-a031-640623bd4272-istio-data\") pod \"e9d98901-eb52-4bd0-a031-640623bd4272\" (UID: \"e9d98901-eb52-4bd0-a031-640623bd4272\") " Nov 04 03:41:55.992673 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:55.992609 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9d98901-eb52-4bd0-a031-640623bd4272-kube-api-access-7pp95\" (UniqueName: \"kubernetes.io/projected/e9d98901-eb52-4bd0-a031-640623bd4272-kube-api-access-7pp95\") pod \"e9d98901-eb52-4bd0-a031-640623bd4272\" (UID: \"e9d98901-eb52-4bd0-a031-640623bd4272\") " Nov 04 03:41:55.992880 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:55.992675 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9d98901-eb52-4bd0-a031-640623bd4272-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9d98901-eb52-4bd0-a031-640623bd4272-workload-certs\") pod \"e9d98901-eb52-4bd0-a031-640623bd4272\" (UID: \"e9d98901-eb52-4bd0-a031-640623bd4272\") " Nov 04 03:41:55.992880 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:55.992692 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9d98901-eb52-4bd0-a031-640623bd4272-istio-token\" (UniqueName: \"kubernetes.io/projected/e9d98901-eb52-4bd0-a031-640623bd4272-istio-token\") pod \"e9d98901-eb52-4bd0-a031-640623bd4272\" (UID: \"e9d98901-eb52-4bd0-a031-640623bd4272\") " Nov 04 03:41:55.992880 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:55.992837 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9d98901-eb52-4bd0-a031-640623bd4272-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9d98901-eb52-4bd0-a031-640623bd4272-istiod-ca-cert\") pod \"e9d98901-eb52-4bd0-a031-640623bd4272\" (UID: \"e9d98901-eb52-4bd0-a031-640623bd4272\") " Nov 04 03:41:55.993070 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:55.992959 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9d98901-eb52-4bd0-a031-640623bd4272-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9d98901-eb52-4bd0-a031-640623bd4272-credential-socket\") pod \"e9d98901-eb52-4bd0-a031-640623bd4272\" (UID: \"e9d98901-eb52-4bd0-a031-640623bd4272\") " Nov 04 03:41:55.993070 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:55.992971 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9d98901-eb52-4bd0-a031-640623bd4272-istio-data" pod "e9d98901-eb52-4bd0-a031-640623bd4272" (UID: "e9d98901-eb52-4bd0-a031-640623bd4272"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:55.993070 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:55.992987 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9d98901-eb52-4bd0-a031-640623bd4272-workload-certs" pod "e9d98901-eb52-4bd0-a031-640623bd4272" (UID: "e9d98901-eb52-4bd0-a031-640623bd4272"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:55.993070 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:55.993011 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9d98901-eb52-4bd0-a031-640623bd4272-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9d98901-eb52-4bd0-a031-640623bd4272-workload-socket\") pod \"e9d98901-eb52-4bd0-a031-640623bd4272\" (UID: \"e9d98901-eb52-4bd0-a031-640623bd4272\") " Nov 04 03:41:55.993309 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:55.993187 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9d98901-eb52-4bd0-a031-640623bd4272-credential-socket" pod "e9d98901-eb52-4bd0-a031-640623bd4272" (UID: "e9d98901-eb52-4bd0-a031-640623bd4272"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:55.993394 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:55.993363 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9d98901-eb52-4bd0-a031-640623bd4272-workload-socket" pod "e9d98901-eb52-4bd0-a031-640623bd4272" (UID: "e9d98901-eb52-4bd0-a031-640623bd4272"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:55.993454 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:55.993389 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9d98901-eb52-4bd0-a031-640623bd4272-istiod-ca-cert" pod "e9d98901-eb52-4bd0-a031-640623bd4272" (UID: "e9d98901-eb52-4bd0-a031-640623bd4272"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:41:55.993558 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:55.993538 5953 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9d98901-eb52-4bd0-a031-640623bd4272-workload-socket\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:55.993643 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:55.993566 5953 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9d98901-eb52-4bd0-a031-640623bd4272-istio-data\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:55.993643 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:55.993581 5953 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9d98901-eb52-4bd0-a031-640623bd4272-workload-certs\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:55.993643 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:55.993594 5953 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9d98901-eb52-4bd0-a031-640623bd4272-istiod-ca-cert\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:55.993759 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:55.993609 5953 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9d98901-eb52-4bd0-a031-640623bd4272-credential-socket\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:55.995406 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:55.995379 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9d98901-eb52-4bd0-a031-640623bd4272-istio-podinfo" pod "e9d98901-eb52-4bd0-a031-640623bd4272" (UID: "e9d98901-eb52-4bd0-a031-640623bd4272"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:41:55.995406 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:55.995386 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9d98901-eb52-4bd0-a031-640623bd4272-istio-envoy" pod "e9d98901-eb52-4bd0-a031-640623bd4272" (UID: "e9d98901-eb52-4bd0-a031-640623bd4272"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:55.995686 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:55.995669 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9d98901-eb52-4bd0-a031-640623bd4272-kube-api-access-7pp95" pod "e9d98901-eb52-4bd0-a031-640623bd4272" (UID: "e9d98901-eb52-4bd0-a031-640623bd4272"). InnerVolumeSpecName "kube-api-access-7pp95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:41:55.995765 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:55.995690 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9d98901-eb52-4bd0-a031-640623bd4272-istio-token" pod "e9d98901-eb52-4bd0-a031-640623bd4272" (UID: "e9d98901-eb52-4bd0-a031-640623bd4272"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:41:56.094732 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:56.094609 5953 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pp95\" (UniqueName: \"kubernetes.io/projected/e9d98901-eb52-4bd0-a031-640623bd4272-kube-api-access-7pp95\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:56.094732 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:56.094675 5953 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9d98901-eb52-4bd0-a031-640623bd4272-istio-token\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:56.094732 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:56.094685 5953 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9d98901-eb52-4bd0-a031-640623bd4272-istio-envoy\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:56.094732 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:56.094694 5953 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9d98901-eb52-4bd0-a031-640623bd4272-istio-podinfo\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:56.531997 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:56.531962 5953 generic.go:356] "Generic (PLEG): container finished" podID="e9d98901-eb52-4bd0-a031-640623bd4272" containerID="631110178fbc6ef1a26586f6ca6109f7d52e464849a0ed7729fe4bf127a1a46b" exitCode=0 Nov 04 03:41:56.532417 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:56.532014 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" Nov 04 03:41:56.532417 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:56.532043 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" event={"ID":"e9d98901-eb52-4bd0-a031-640623bd4272","Type":"ContainerDied","Data":"631110178fbc6ef1a26586f6ca6109f7d52e464849a0ed7729fe4bf127a1a46b"} Nov 04 03:41:56.532417 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:56.532097 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8" event={"ID":"e9d98901-eb52-4bd0-a031-640623bd4272","Type":"ContainerDied","Data":"8c7263742c8eebe1542edf5f17c0d8d3891ce97419bae31b7b268a34898ef5f1"} Nov 04 03:41:56.532417 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:56.532119 5953 scope.go:122] "RemoveContainer" containerID="631110178fbc6ef1a26586f6ca6109f7d52e464849a0ed7729fe4bf127a1a46b" Nov 04 03:41:56.541970 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:56.541955 5953 scope.go:122] "RemoveContainer" containerID="9494120f489a4d24f206df071b044ca6655db40dd6f4ac65f423685e8added78" Nov 04 03:41:56.550840 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:56.550821 5953 scope.go:122] "RemoveContainer" containerID="631110178fbc6ef1a26586f6ca6109f7d52e464849a0ed7729fe4bf127a1a46b" Nov 04 03:41:56.551150 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:41:56.551125 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"631110178fbc6ef1a26586f6ca6109f7d52e464849a0ed7729fe4bf127a1a46b\": container with ID starting with 631110178fbc6ef1a26586f6ca6109f7d52e464849a0ed7729fe4bf127a1a46b not found: ID does not exist" containerID="631110178fbc6ef1a26586f6ca6109f7d52e464849a0ed7729fe4bf127a1a46b" Nov 04 03:41:56.551215 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:56.551159 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"631110178fbc6ef1a26586f6ca6109f7d52e464849a0ed7729fe4bf127a1a46b"} err="failed to get container status \"631110178fbc6ef1a26586f6ca6109f7d52e464849a0ed7729fe4bf127a1a46b\": rpc error: code = NotFound desc = could not find container \"631110178fbc6ef1a26586f6ca6109f7d52e464849a0ed7729fe4bf127a1a46b\": container with ID starting with 631110178fbc6ef1a26586f6ca6109f7d52e464849a0ed7729fe4bf127a1a46b not found: ID does not exist" Nov 04 03:41:56.551215 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:56.551181 5953 scope.go:122] "RemoveContainer" containerID="9494120f489a4d24f206df071b044ca6655db40dd6f4ac65f423685e8added78" Nov 04 03:41:56.551432 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:41:56.551412 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9494120f489a4d24f206df071b044ca6655db40dd6f4ac65f423685e8added78\": container with ID starting with 9494120f489a4d24f206df071b044ca6655db40dd6f4ac65f423685e8added78 not found: ID does not exist" containerID="9494120f489a4d24f206df071b044ca6655db40dd6f4ac65f423685e8added78" Nov 04 03:41:56.551478 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:56.551439 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9494120f489a4d24f206df071b044ca6655db40dd6f4ac65f423685e8added78"} err="failed to get container status \"9494120f489a4d24f206df071b044ca6655db40dd6f4ac65f423685e8added78\": rpc error: code = NotFound desc = could not find container \"9494120f489a4d24f206df071b044ca6655db40dd6f4ac65f423685e8added78\": container with ID starting with 9494120f489a4d24f206df071b044ca6655db40dd6f4ac65f423685e8added78 not found: ID does not exist" Nov 04 03:41:56.748875 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:56.748843 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8"] Nov 04 03:41:56.799252 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:56.799172 5953 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-pnzc8"] Nov 04 03:41:56.862410 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:56.862378 5953 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9d98901-eb52-4bd0-a031-640623bd4272" path="/var/lib/kubelet/pods/e9d98901-eb52-4bd0-a031-640623bd4272/volumes" Nov 04 03:41:59.553404 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:59.553364 5953 generic.go:356] "Generic (PLEG): container finished" podID="e573d41f-40a7-4e92-880d-a0204b14633a" containerID="e02083754d8c50a5bf60ad89092401202602e4d2e292bf9ad57b909627192e31" exitCode=0 Nov 04 03:41:59.553804 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:59.553438 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" event={"ID":"e573d41f-40a7-4e92-880d-a0204b14633a","Type":"ContainerDied","Data":"e02083754d8c50a5bf60ad89092401202602e4d2e292bf9ad57b909627192e31"} Nov 04 03:41:59.619401 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:59.619330 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" Nov 04 03:41:59.736637 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:59.736579 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e573d41f-40a7-4e92-880d-a0204b14633a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e573d41f-40a7-4e92-880d-a0204b14633a-istio-envoy\") pod \"e573d41f-40a7-4e92-880d-a0204b14633a\" (UID: \"e573d41f-40a7-4e92-880d-a0204b14633a\") " Nov 04 03:41:59.736841 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:59.736661 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e573d41f-40a7-4e92-880d-a0204b14633a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e573d41f-40a7-4e92-880d-a0204b14633a-credential-socket\") pod \"e573d41f-40a7-4e92-880d-a0204b14633a\" (UID: \"e573d41f-40a7-4e92-880d-a0204b14633a\") " Nov 04 03:41:59.736841 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:59.736728 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e573d41f-40a7-4e92-880d-a0204b14633a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e573d41f-40a7-4e92-880d-a0204b14633a-workload-socket\") pod \"e573d41f-40a7-4e92-880d-a0204b14633a\" (UID: \"e573d41f-40a7-4e92-880d-a0204b14633a\") " Nov 04 03:41:59.736841 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:59.736758 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e573d41f-40a7-4e92-880d-a0204b14633a-istio-token\" (UniqueName: \"kubernetes.io/projected/e573d41f-40a7-4e92-880d-a0204b14633a-istio-token\") pod \"e573d41f-40a7-4e92-880d-a0204b14633a\" (UID: \"e573d41f-40a7-4e92-880d-a0204b14633a\") " Nov 04 03:41:59.736841 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:59.736827 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e573d41f-40a7-4e92-880d-a0204b14633a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e573d41f-40a7-4e92-880d-a0204b14633a-istiod-ca-cert\") pod \"e573d41f-40a7-4e92-880d-a0204b14633a\" (UID: \"e573d41f-40a7-4e92-880d-a0204b14633a\") " Nov 04 03:41:59.737073 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:59.736874 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e573d41f-40a7-4e92-880d-a0204b14633a-kube-api-access-27s7q\" (UniqueName: \"kubernetes.io/projected/e573d41f-40a7-4e92-880d-a0204b14633a-kube-api-access-27s7q\") pod \"e573d41f-40a7-4e92-880d-a0204b14633a\" (UID: \"e573d41f-40a7-4e92-880d-a0204b14633a\") " Nov 04 03:41:59.737073 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:59.736933 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e573d41f-40a7-4e92-880d-a0204b14633a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e573d41f-40a7-4e92-880d-a0204b14633a-istio-data\") pod \"e573d41f-40a7-4e92-880d-a0204b14633a\" (UID: \"e573d41f-40a7-4e92-880d-a0204b14633a\") " Nov 04 03:41:59.737073 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:59.737009 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e573d41f-40a7-4e92-880d-a0204b14633a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e573d41f-40a7-4e92-880d-a0204b14633a-workload-certs\") pod \"e573d41f-40a7-4e92-880d-a0204b14633a\" (UID: \"e573d41f-40a7-4e92-880d-a0204b14633a\") " Nov 04 03:41:59.737073 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:59.737028 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e573d41f-40a7-4e92-880d-a0204b14633a-workload-socket" pod "e573d41f-40a7-4e92-880d-a0204b14633a" (UID: "e573d41f-40a7-4e92-880d-a0204b14633a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:59.737073 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:59.737061 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e573d41f-40a7-4e92-880d-a0204b14633a-credential-socket" pod "e573d41f-40a7-4e92-880d-a0204b14633a" (UID: "e573d41f-40a7-4e92-880d-a0204b14633a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:59.737311 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:59.737171 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e573d41f-40a7-4e92-880d-a0204b14633a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e573d41f-40a7-4e92-880d-a0204b14633a-istio-podinfo\") pod \"e573d41f-40a7-4e92-880d-a0204b14633a\" (UID: \"e573d41f-40a7-4e92-880d-a0204b14633a\") " Nov 04 03:41:59.737311 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:59.737204 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e573d41f-40a7-4e92-880d-a0204b14633a-istio-data" pod "e573d41f-40a7-4e92-880d-a0204b14633a" (UID: "e573d41f-40a7-4e92-880d-a0204b14633a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:59.737311 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:59.737262 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e573d41f-40a7-4e92-880d-a0204b14633a-workload-certs" pod "e573d41f-40a7-4e92-880d-a0204b14633a" (UID: "e573d41f-40a7-4e92-880d-a0204b14633a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:59.737603 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:59.737584 5953 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e573d41f-40a7-4e92-880d-a0204b14633a-credential-socket\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:59.737698 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:59.737608 5953 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e573d41f-40a7-4e92-880d-a0204b14633a-workload-socket\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:59.737698 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:59.737647 5953 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e573d41f-40a7-4e92-880d-a0204b14633a-istio-data\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:59.737698 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:59.737661 5953 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e573d41f-40a7-4e92-880d-a0204b14633a-workload-certs\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:59.738101 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:59.738077 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e573d41f-40a7-4e92-880d-a0204b14633a-istiod-ca-cert" pod "e573d41f-40a7-4e92-880d-a0204b14633a" (UID: "e573d41f-40a7-4e92-880d-a0204b14633a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:41:59.739432 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:59.739412 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e573d41f-40a7-4e92-880d-a0204b14633a-istio-podinfo" pod "e573d41f-40a7-4e92-880d-a0204b14633a" (UID: "e573d41f-40a7-4e92-880d-a0204b14633a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:41:59.739519 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:59.739494 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e573d41f-40a7-4e92-880d-a0204b14633a-kube-api-access-27s7q" pod "e573d41f-40a7-4e92-880d-a0204b14633a" (UID: "e573d41f-40a7-4e92-880d-a0204b14633a"). InnerVolumeSpecName "kube-api-access-27s7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:41:59.739612 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:59.739596 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e573d41f-40a7-4e92-880d-a0204b14633a-istio-envoy" pod "e573d41f-40a7-4e92-880d-a0204b14633a" (UID: "e573d41f-40a7-4e92-880d-a0204b14633a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:59.739692 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:59.739670 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e573d41f-40a7-4e92-880d-a0204b14633a-istio-token" pod "e573d41f-40a7-4e92-880d-a0204b14633a" (UID: "e573d41f-40a7-4e92-880d-a0204b14633a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:41:59.839291 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:59.839257 5953 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e573d41f-40a7-4e92-880d-a0204b14633a-istio-podinfo\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:59.839291 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:59.839288 5953 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e573d41f-40a7-4e92-880d-a0204b14633a-istio-envoy\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:59.839502 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:59.839301 5953 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e573d41f-40a7-4e92-880d-a0204b14633a-istio-token\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:59.839502 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:59.839314 5953 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e573d41f-40a7-4e92-880d-a0204b14633a-istiod-ca-cert\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:59.839502 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:41:59.839329 5953 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27s7q\" (UniqueName: \"kubernetes.io/projected/e573d41f-40a7-4e92-880d-a0204b14633a-kube-api-access-27s7q\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:42:00.559642 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:00.559598 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" event={"ID":"e573d41f-40a7-4e92-880d-a0204b14633a","Type":"ContainerDied","Data":"0c458d67f8a3d2a25e336eeb8f3832cb691ff72007a164387aa89d6f791d474d"} Nov 04 03:42:00.560025 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:00.559690 5953 scope.go:122] "RemoveContainer" containerID="e02083754d8c50a5bf60ad89092401202602e4d2e292bf9ad57b909627192e31" Nov 04 03:42:00.560025 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:00.559633 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w" Nov 04 03:42:00.725681 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:00.725647 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w"] Nov 04 03:42:00.776172 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:00.776147 5953 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-nd22w"] Nov 04 03:42:00.794614 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:00.794547 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" Nov 04 03:42:00.850859 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:00.850830 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/745d29e4-d827-4dc9-b223-2e1f345518ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/745d29e4-d827-4dc9-b223-2e1f345518ef-workload-socket\") pod \"745d29e4-d827-4dc9-b223-2e1f345518ef\" (UID: \"745d29e4-d827-4dc9-b223-2e1f345518ef\") " Nov 04 03:42:00.851056 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:00.850897 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/745d29e4-d827-4dc9-b223-2e1f345518ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/745d29e4-d827-4dc9-b223-2e1f345518ef-istio-podinfo\") pod \"745d29e4-d827-4dc9-b223-2e1f345518ef\" (UID: \"745d29e4-d827-4dc9-b223-2e1f345518ef\") " Nov 04 03:42:00.851056 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:00.850927 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/745d29e4-d827-4dc9-b223-2e1f345518ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/745d29e4-d827-4dc9-b223-2e1f345518ef-istio-data\") pod \"745d29e4-d827-4dc9-b223-2e1f345518ef\" (UID: \"745d29e4-d827-4dc9-b223-2e1f345518ef\") " Nov 04 03:42:00.851056 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:00.850992 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/745d29e4-d827-4dc9-b223-2e1f345518ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/745d29e4-d827-4dc9-b223-2e1f345518ef-istiod-ca-cert\") pod \"745d29e4-d827-4dc9-b223-2e1f345518ef\" (UID: \"745d29e4-d827-4dc9-b223-2e1f345518ef\") " Nov 04 03:42:00.851056 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:00.851035 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/745d29e4-d827-4dc9-b223-2e1f345518ef-istio-token\" (UniqueName: \"kubernetes.io/projected/745d29e4-d827-4dc9-b223-2e1f345518ef-istio-token\") pod \"745d29e4-d827-4dc9-b223-2e1f345518ef\" (UID: \"745d29e4-d827-4dc9-b223-2e1f345518ef\") " Nov 04 03:42:00.851269 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:00.851108 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/745d29e4-d827-4dc9-b223-2e1f345518ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/745d29e4-d827-4dc9-b223-2e1f345518ef-workload-certs\") pod \"745d29e4-d827-4dc9-b223-2e1f345518ef\" (UID: \"745d29e4-d827-4dc9-b223-2e1f345518ef\") " Nov 04 03:42:00.851269 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:00.851122 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/745d29e4-d827-4dc9-b223-2e1f345518ef-workload-socket" pod "745d29e4-d827-4dc9-b223-2e1f345518ef" (UID: "745d29e4-d827-4dc9-b223-2e1f345518ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:42:00.851269 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:00.851164 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/745d29e4-d827-4dc9-b223-2e1f345518ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/745d29e4-d827-4dc9-b223-2e1f345518ef-credential-socket\") pod \"745d29e4-d827-4dc9-b223-2e1f345518ef\" (UID: \"745d29e4-d827-4dc9-b223-2e1f345518ef\") " Nov 04 03:42:00.851269 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:00.851227 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/745d29e4-d827-4dc9-b223-2e1f345518ef-kube-api-access-6kzb5\" (UniqueName: \"kubernetes.io/projected/745d29e4-d827-4dc9-b223-2e1f345518ef-kube-api-access-6kzb5\") pod \"745d29e4-d827-4dc9-b223-2e1f345518ef\" (UID: \"745d29e4-d827-4dc9-b223-2e1f345518ef\") " Nov 04 03:42:00.851269 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:00.851261 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/745d29e4-d827-4dc9-b223-2e1f345518ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/745d29e4-d827-4dc9-b223-2e1f345518ef-istio-envoy\") pod \"745d29e4-d827-4dc9-b223-2e1f345518ef\" (UID: \"745d29e4-d827-4dc9-b223-2e1f345518ef\") " Nov 04 03:42:00.851528 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:00.851403 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/745d29e4-d827-4dc9-b223-2e1f345518ef-workload-certs" pod "745d29e4-d827-4dc9-b223-2e1f345518ef" (UID: "745d29e4-d827-4dc9-b223-2e1f345518ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:42:00.851528 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:00.851439 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/745d29e4-d827-4dc9-b223-2e1f345518ef-istiod-ca-cert" pod "745d29e4-d827-4dc9-b223-2e1f345518ef" (UID: "745d29e4-d827-4dc9-b223-2e1f345518ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:42:00.851664 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:00.851614 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/745d29e4-d827-4dc9-b223-2e1f345518ef-credential-socket" pod "745d29e4-d827-4dc9-b223-2e1f345518ef" (UID: "745d29e4-d827-4dc9-b223-2e1f345518ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:42:00.852102 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:00.852008 5953 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/745d29e4-d827-4dc9-b223-2e1f345518ef-credential-socket\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:42:00.852102 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:00.852037 5953 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/745d29e4-d827-4dc9-b223-2e1f345518ef-workload-socket\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:42:00.852102 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:00.852053 5953 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/745d29e4-d827-4dc9-b223-2e1f345518ef-istiod-ca-cert\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:42:00.852102 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:00.852070 5953 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/745d29e4-d827-4dc9-b223-2e1f345518ef-workload-certs\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:42:00.852477 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:00.852281 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/745d29e4-d827-4dc9-b223-2e1f345518ef-istio-data" pod "745d29e4-d827-4dc9-b223-2e1f345518ef" (UID: "745d29e4-d827-4dc9-b223-2e1f345518ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:42:00.853544 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:00.853518 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/745d29e4-d827-4dc9-b223-2e1f345518ef-istio-podinfo" pod "745d29e4-d827-4dc9-b223-2e1f345518ef" (UID: "745d29e4-d827-4dc9-b223-2e1f345518ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:42:00.853734 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:00.853712 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/745d29e4-d827-4dc9-b223-2e1f345518ef-istio-envoy" pod "745d29e4-d827-4dc9-b223-2e1f345518ef" (UID: "745d29e4-d827-4dc9-b223-2e1f345518ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:42:00.853823 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:00.853799 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/745d29e4-d827-4dc9-b223-2e1f345518ef-kube-api-access-6kzb5" pod "745d29e4-d827-4dc9-b223-2e1f345518ef" (UID: "745d29e4-d827-4dc9-b223-2e1f345518ef"). InnerVolumeSpecName "kube-api-access-6kzb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:42:00.853877 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:00.853838 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/745d29e4-d827-4dc9-b223-2e1f345518ef-istio-token" pod "745d29e4-d827-4dc9-b223-2e1f345518ef" (UID: "745d29e4-d827-4dc9-b223-2e1f345518ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:42:00.857959 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:00.857929 5953 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e573d41f-40a7-4e92-880d-a0204b14633a" path="/var/lib/kubelet/pods/e573d41f-40a7-4e92-880d-a0204b14633a/volumes" Nov 04 03:42:00.953190 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:00.953152 5953 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kzb5\" (UniqueName: \"kubernetes.io/projected/745d29e4-d827-4dc9-b223-2e1f345518ef-kube-api-access-6kzb5\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:42:00.953190 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:00.953184 5953 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/745d29e4-d827-4dc9-b223-2e1f345518ef-istio-envoy\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:42:00.953190 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:00.953197 5953 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/745d29e4-d827-4dc9-b223-2e1f345518ef-istio-podinfo\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:42:00.953504 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:00.953212 5953 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/745d29e4-d827-4dc9-b223-2e1f345518ef-istio-data\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:42:00.953504 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:00.953224 5953 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/745d29e4-d827-4dc9-b223-2e1f345518ef-istio-token\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:42:01.567721 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:01.567679 5953 generic.go:356] "Generic (PLEG): container finished" podID="745d29e4-d827-4dc9-b223-2e1f345518ef" containerID="97d4ceae7ab18a1b9ac458d23a07d31c4b351f9246767a755e044b609c02177d" exitCode=0 Nov 04 03:42:01.568142 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:01.567752 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" event={"ID":"745d29e4-d827-4dc9-b223-2e1f345518ef","Type":"ContainerDied","Data":"97d4ceae7ab18a1b9ac458d23a07d31c4b351f9246767a755e044b609c02177d"} Nov 04 03:42:01.568142 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:01.567818 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" event={"ID":"745d29e4-d827-4dc9-b223-2e1f345518ef","Type":"ContainerDied","Data":"c8912f0d3c2a583a637a05f780801afbfcacee219f6d89274f53d3b0f859b04e"} Nov 04 03:42:01.568142 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:01.567755 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw" Nov 04 03:42:01.568142 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:01.567846 5953 scope.go:122] "RemoveContainer" containerID="97d4ceae7ab18a1b9ac458d23a07d31c4b351f9246767a755e044b609c02177d" Nov 04 03:42:01.577004 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:01.576985 5953 scope.go:122] "RemoveContainer" containerID="97d4ceae7ab18a1b9ac458d23a07d31c4b351f9246767a755e044b609c02177d" Nov 04 03:42:01.577312 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:42:01.577291 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97d4ceae7ab18a1b9ac458d23a07d31c4b351f9246767a755e044b609c02177d\": container with ID starting with 97d4ceae7ab18a1b9ac458d23a07d31c4b351f9246767a755e044b609c02177d not found: ID does not exist" containerID="97d4ceae7ab18a1b9ac458d23a07d31c4b351f9246767a755e044b609c02177d" Nov 04 03:42:01.577366 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:01.577322 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97d4ceae7ab18a1b9ac458d23a07d31c4b351f9246767a755e044b609c02177d"} err="failed to get container status \"97d4ceae7ab18a1b9ac458d23a07d31c4b351f9246767a755e044b609c02177d\": rpc error: code = NotFound desc = could not find container \"97d4ceae7ab18a1b9ac458d23a07d31c4b351f9246767a755e044b609c02177d\": container with ID starting with 97d4ceae7ab18a1b9ac458d23a07d31c4b351f9246767a755e044b609c02177d not found: ID does not exist" Nov 04 03:42:01.758601 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:01.758562 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw"] Nov 04 03:42:01.807837 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:01.807805 5953 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-rrnsw"] Nov 04 03:42:02.856595 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:42:02.856565 5953 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="745d29e4-d827-4dc9-b223-2e1f345518ef" path="/var/lib/kubelet/pods/745d29e4-d827-4dc9-b223-2e1f345518ef/volumes" Nov 04 03:42:16.384797 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:42:16.384720 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:42:46.384750 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:42:46.384711 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:43:16.384875 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:43:16.384839 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:43:46.384957 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:43:46.384916 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:44:16.384464 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:44:16.384417 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:44:46.385136 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:44:46.385093 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:45:00.183948 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:00.183908 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896065-77nkm"] Nov 04 03:45:00.186466 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:00.184580 5953 cpu_manager.go:414] "containerMap: removing container" podUID="e573d41f-40a7-4e92-880d-a0204b14633a" containerName="istio-proxy" Nov 04 03:45:00.186466 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:00.184596 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e573d41f-40a7-4e92-880d-a0204b14633a" containerName="istio-proxy" Nov 04 03:45:00.186466 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:00.184636 5953 cpu_manager.go:414] "containerMap: removing container" podUID="e9d98901-eb52-4bd0-a031-640623bd4272" containerName="istio-proxy" Nov 04 03:45:00.186466 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:00.184642 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9d98901-eb52-4bd0-a031-640623bd4272" containerName="istio-proxy" Nov 04 03:45:00.186466 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:00.184665 5953 cpu_manager.go:414] "containerMap: removing container" podUID="e9d98901-eb52-4bd0-a031-640623bd4272" containerName="istio-proxy" Nov 04 03:45:00.186466 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:00.184670 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9d98901-eb52-4bd0-a031-640623bd4272" containerName="istio-proxy" Nov 04 03:45:00.186466 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:00.184678 5953 cpu_manager.go:414] "containerMap: removing container" podUID="8772665e-c7a2-4823-be72-2b93863039ed" containerName="payload-pre-processing" Nov 04 03:45:00.186466 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:00.184683 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8772665e-c7a2-4823-be72-2b93863039ed" containerName="payload-pre-processing" Nov 04 03:45:00.186466 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:00.184691 5953 cpu_manager.go:414] "containerMap: removing container" podUID="745d29e4-d827-4dc9-b223-2e1f345518ef" containerName="istio-proxy" Nov 04 03:45:00.186466 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:00.184696 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="745d29e4-d827-4dc9-b223-2e1f345518ef" containerName="istio-proxy" Nov 04 03:45:00.186466 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:00.184762 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="587e29e9-ccc7-4343-8dd3-2fd2088f5d6c" containerName="istio-proxy" Nov 04 03:45:00.186466 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:00.184772 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9d98901-eb52-4bd0-a031-640623bd4272" containerName="istio-proxy" Nov 04 03:45:00.186466 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:00.184778 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="8772665e-c7a2-4823-be72-2b93863039ed" containerName="payload-pre-processing" Nov 04 03:45:00.186466 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:00.184784 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="745d29e4-d827-4dc9-b223-2e1f345518ef" containerName="istio-proxy" Nov 04 03:45:00.186466 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:00.184794 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="e573d41f-40a7-4e92-880d-a0204b14633a" containerName="istio-proxy" Nov 04 03:45:00.186466 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:00.184802 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="48e61829-0b7d-40a9-b877-398aa633b44f" containerName="istio-proxy" Nov 04 03:45:00.187825 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:00.187807 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896065-77nkm" Nov 04 03:45:00.239818 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:00.239788 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9mph\" (UniqueName: \"kubernetes.io/projected/020ca3c3-bc40-4131-8e81-6f4bb1ffefdb-kube-api-access-l9mph\") pod \"maas-api-key-cleanup-29896065-77nkm\" (UID: \"020ca3c3-bc40-4131-8e81-6f4bb1ffefdb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896065-77nkm" Nov 04 03:45:00.240033 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:00.239975 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/020ca3c3-bc40-4131-8e81-6f4bb1ffefdb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29896065-77nkm\" (UID: \"020ca3c3-bc40-4131-8e81-6f4bb1ffefdb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896065-77nkm" Nov 04 03:45:00.242897 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:00.242873 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hlrnm\"" Nov 04 03:45:00.296779 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:00.296744 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896065-77nkm"] Nov 04 03:45:00.341198 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:00.341167 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9mph\" (UniqueName: \"kubernetes.io/projected/020ca3c3-bc40-4131-8e81-6f4bb1ffefdb-kube-api-access-l9mph\") pod \"maas-api-key-cleanup-29896065-77nkm\" (UID: \"020ca3c3-bc40-4131-8e81-6f4bb1ffefdb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896065-77nkm" Nov 04 03:45:00.341382 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:00.341295 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/020ca3c3-bc40-4131-8e81-6f4bb1ffefdb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29896065-77nkm\" (UID: \"020ca3c3-bc40-4131-8e81-6f4bb1ffefdb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896065-77nkm" Nov 04 03:45:00.341921 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:00.341899 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/020ca3c3-bc40-4131-8e81-6f4bb1ffefdb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29896065-77nkm\" (UID: \"020ca3c3-bc40-4131-8e81-6f4bb1ffefdb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896065-77nkm" Nov 04 03:45:00.402950 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:00.402918 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9mph\" (UniqueName: \"kubernetes.io/projected/020ca3c3-bc40-4131-8e81-6f4bb1ffefdb-kube-api-access-l9mph\") pod \"maas-api-key-cleanup-29896065-77nkm\" (UID: \"020ca3c3-bc40-4131-8e81-6f4bb1ffefdb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896065-77nkm" Nov 04 03:45:00.501852 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:00.501827 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896065-77nkm" Nov 04 03:45:00.874419 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:00.874279 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896065-77nkm"] Nov 04 03:45:00.877310 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:45:00.877282 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod020ca3c3_bc40_4131_8e81_6f4bb1ffefdb.slice/crio-fd3830cd32d3f7931f4b5e7713bf462d3b0213d9ff7abd40c9a21dab005e4151 WatchSource:0}: Error finding container fd3830cd32d3f7931f4b5e7713bf462d3b0213d9ff7abd40c9a21dab005e4151: Status 404 returned error can't find the container with id fd3830cd32d3f7931f4b5e7713bf462d3b0213d9ff7abd40c9a21dab005e4151 Nov 04 03:45:01.537366 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:01.537327 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896065-77nkm" event={"ID":"020ca3c3-bc40-4131-8e81-6f4bb1ffefdb","Type":"ContainerStarted","Data":"f8346fdfe13b6e0d4ca7385714fc8b65a963f9b874c542e525aaaf498235941c"} Nov 04 03:45:01.537810 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:01.537383 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896065-77nkm" event={"ID":"020ca3c3-bc40-4131-8e81-6f4bb1ffefdb","Type":"ContainerStarted","Data":"fd3830cd32d3f7931f4b5e7713bf462d3b0213d9ff7abd40c9a21dab005e4151"} Nov 04 03:45:01.655508 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:01.655451 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896065-77nkm" podStartSLOduration=1.655437 podStartE2EDuration="1.655437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:45:01.653494639 +0000 UTC m=+1626.928125402" watchObservedRunningTime="2026-11-04 03:45:01.655437 +0000 UTC m=+1626.930067776" Nov 04 03:45:02.543539 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:02.543504 5953 generic.go:356] "Generic (PLEG): container finished" podID="020ca3c3-bc40-4131-8e81-6f4bb1ffefdb" containerID="f8346fdfe13b6e0d4ca7385714fc8b65a963f9b874c542e525aaaf498235941c" exitCode=60 Nov 04 03:45:02.543993 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:02.543594 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896065-77nkm" event={"ID":"020ca3c3-bc40-4131-8e81-6f4bb1ffefdb","Type":"ContainerDied","Data":"f8346fdfe13b6e0d4ca7385714fc8b65a963f9b874c542e525aaaf498235941c"} Nov 04 03:45:02.543993 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:02.543886 5953 scope.go:122] "RemoveContainer" containerID="f8346fdfe13b6e0d4ca7385714fc8b65a963f9b874c542e525aaaf498235941c" Nov 04 03:45:03.550937 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:03.550896 5953 generic.go:356] "Generic (PLEG): container finished" podID="020ca3c3-bc40-4131-8e81-6f4bb1ffefdb" containerID="068dce81de4644b4d24e9cf28f1c01fc05ed612a0abbd86df0bf4db02ab77bb5" exitCode=60 Nov 04 03:45:03.551435 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:03.550947 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896065-77nkm" event={"ID":"020ca3c3-bc40-4131-8e81-6f4bb1ffefdb","Type":"ContainerDied","Data":"068dce81de4644b4d24e9cf28f1c01fc05ed612a0abbd86df0bf4db02ab77bb5"} Nov 04 03:45:03.551435 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:03.551005 5953 scope.go:122] "RemoveContainer" containerID="f8346fdfe13b6e0d4ca7385714fc8b65a963f9b874c542e525aaaf498235941c" Nov 04 03:45:03.551435 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:03.551293 5953 scope.go:122] "RemoveContainer" containerID="068dce81de4644b4d24e9cf28f1c01fc05ed612a0abbd86df0bf4db02ab77bb5" Nov 04 03:45:03.551650 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:45:03.551601 5953 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29896065-77nkm_odh-ai-gateway-infra(020ca3c3-bc40-4131-8e81-6f4bb1ffefdb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896065-77nkm" podUID="020ca3c3-bc40-4131-8e81-6f4bb1ffefdb" Nov 04 03:45:04.558859 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:04.558830 5953 scope.go:122] "RemoveContainer" containerID="068dce81de4644b4d24e9cf28f1c01fc05ed612a0abbd86df0bf4db02ab77bb5" Nov 04 03:45:04.559266 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:45:04.559074 5953 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29896065-77nkm_odh-ai-gateway-infra(020ca3c3-bc40-4131-8e81-6f4bb1ffefdb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896065-77nkm" podUID="020ca3c3-bc40-4131-8e81-6f4bb1ffefdb" Nov 04 03:45:15.849197 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:15.849168 5953 scope.go:122] "RemoveContainer" containerID="068dce81de4644b4d24e9cf28f1c01fc05ed612a0abbd86df0bf4db02ab77bb5" Nov 04 03:45:16.384565 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:45:16.384472 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:45:16.622284 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:16.622255 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896065-77nkm" event={"ID":"020ca3c3-bc40-4131-8e81-6f4bb1ffefdb","Type":"ContainerStarted","Data":"a607a5081dd03816d2f4fddd18d7e4aabf899cb794d39c68e6ac837e1931be52"} Nov 04 03:45:16.622704 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:16.622684 5953 scope.go:122] "RemoveContainer" containerID="a607a5081dd03816d2f4fddd18d7e4aabf899cb794d39c68e6ac837e1931be52" Nov 04 03:45:16.622976 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:45:16.622954 5953 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29896065-77nkm_odh-ai-gateway-infra(020ca3c3-bc40-4131-8e81-6f4bb1ffefdb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896065-77nkm" podUID="020ca3c3-bc40-4131-8e81-6f4bb1ffefdb" Nov 04 03:45:17.630964 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:17.630933 5953 generic.go:356] "Generic (PLEG): container finished" podID="020ca3c3-bc40-4131-8e81-6f4bb1ffefdb" containerID="a607a5081dd03816d2f4fddd18d7e4aabf899cb794d39c68e6ac837e1931be52" exitCode=60 Nov 04 03:45:17.631521 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:17.631003 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896065-77nkm" event={"ID":"020ca3c3-bc40-4131-8e81-6f4bb1ffefdb","Type":"ContainerDied","Data":"a607a5081dd03816d2f4fddd18d7e4aabf899cb794d39c68e6ac837e1931be52"} Nov 04 03:45:17.631521 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:17.631095 5953 scope.go:122] "RemoveContainer" containerID="068dce81de4644b4d24e9cf28f1c01fc05ed612a0abbd86df0bf4db02ab77bb5" Nov 04 03:45:17.631668 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:17.631530 5953 scope.go:122] "RemoveContainer" containerID="a607a5081dd03816d2f4fddd18d7e4aabf899cb794d39c68e6ac837e1931be52" Nov 04 03:45:17.631853 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:45:17.631832 5953 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29896065-77nkm_odh-ai-gateway-infra(020ca3c3-bc40-4131-8e81-6f4bb1ffefdb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896065-77nkm" podUID="020ca3c3-bc40-4131-8e81-6f4bb1ffefdb" Nov 04 03:45:17.748251 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:17.748219 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896065-77nkm"] Nov 04 03:45:18.770159 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:18.770132 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896065-77nkm" Nov 04 03:45:18.846005 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:18.845973 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/020ca3c3-bc40-4131-8e81-6f4bb1ffefdb-kube-api-access-l9mph\" (UniqueName: \"kubernetes.io/projected/020ca3c3-bc40-4131-8e81-6f4bb1ffefdb-kube-api-access-l9mph\") pod \"020ca3c3-bc40-4131-8e81-6f4bb1ffefdb\" (UID: \"020ca3c3-bc40-4131-8e81-6f4bb1ffefdb\") " Nov 04 03:45:18.846178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:18.846078 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/020ca3c3-bc40-4131-8e81-6f4bb1ffefdb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/020ca3c3-bc40-4131-8e81-6f4bb1ffefdb-maas-api-service-ca\") pod \"020ca3c3-bc40-4131-8e81-6f4bb1ffefdb\" (UID: \"020ca3c3-bc40-4131-8e81-6f4bb1ffefdb\") " Nov 04 03:45:18.846436 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:18.846411 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/020ca3c3-bc40-4131-8e81-6f4bb1ffefdb-maas-api-service-ca" pod "020ca3c3-bc40-4131-8e81-6f4bb1ffefdb" (UID: "020ca3c3-bc40-4131-8e81-6f4bb1ffefdb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:45:18.848288 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:18.848266 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/020ca3c3-bc40-4131-8e81-6f4bb1ffefdb-kube-api-access-l9mph" pod "020ca3c3-bc40-4131-8e81-6f4bb1ffefdb" (UID: "020ca3c3-bc40-4131-8e81-6f4bb1ffefdb"). InnerVolumeSpecName "kube-api-access-l9mph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:45:18.946977 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:18.946898 5953 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9mph\" (UniqueName: \"kubernetes.io/projected/020ca3c3-bc40-4131-8e81-6f4bb1ffefdb-kube-api-access-l9mph\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:45:18.946977 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:18.946923 5953 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/020ca3c3-bc40-4131-8e81-6f4bb1ffefdb-maas-api-service-ca\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:45:19.644669 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:19.644633 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896065-77nkm" Nov 04 03:45:19.644669 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:19.644647 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896065-77nkm" event={"ID":"020ca3c3-bc40-4131-8e81-6f4bb1ffefdb","Type":"ContainerDied","Data":"fd3830cd32d3f7931f4b5e7713bf462d3b0213d9ff7abd40c9a21dab005e4151"} Nov 04 03:45:19.644889 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:19.644718 5953 scope.go:122] "RemoveContainer" containerID="a607a5081dd03816d2f4fddd18d7e4aabf899cb794d39c68e6ac837e1931be52" Nov 04 03:45:19.811664 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:19.811628 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896065-77nkm"] Nov 04 03:45:19.861582 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:19.861549 5953 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896065-77nkm"] Nov 04 03:45:20.855666 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:45:20.855608 5953 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="020ca3c3-bc40-4131-8e81-6f4bb1ffefdb" path="/var/lib/kubelet/pods/020ca3c3-bc40-4131-8e81-6f4bb1ffefdb/volumes" Nov 04 03:45:46.385537 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:45:46.385491 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:46:16.385459 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:46:16.385421 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:46:39.629717 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:46:39.629685 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g"] Nov 04 03:46:39.630217 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:46:39.629863 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g" podUID="db0efa5b-3ced-44e4-ba5c-f3bdad377597" containerName="maas-api" containerID="cri-o://c6b09e9da904eaa5e56cf427947508aa0b4f03104393f458ef259e63af30ef36" gracePeriod=30 Nov 04 03:46:39.894189 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:46:39.894125 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g" Nov 04 03:46:39.958611 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:46:39.958578 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db0efa5b-3ced-44e4-ba5c-f3bdad377597-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db0efa5b-3ced-44e4-ba5c-f3bdad377597-openshift-service-ca\") pod \"db0efa5b-3ced-44e4-ba5c-f3bdad377597\" (UID: \"db0efa5b-3ced-44e4-ba5c-f3bdad377597\") " Nov 04 03:46:39.958840 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:46:39.958671 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db0efa5b-3ced-44e4-ba5c-f3bdad377597-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db0efa5b-3ced-44e4-ba5c-f3bdad377597-maas-api-tls\") pod \"db0efa5b-3ced-44e4-ba5c-f3bdad377597\" (UID: \"db0efa5b-3ced-44e4-ba5c-f3bdad377597\") " Nov 04 03:46:39.958840 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:46:39.958762 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db0efa5b-3ced-44e4-ba5c-f3bdad377597-metrics-certs\" (UniqueName: \"kubernetes.io/secret/db0efa5b-3ced-44e4-ba5c-f3bdad377597-metrics-certs\") pod \"db0efa5b-3ced-44e4-ba5c-f3bdad377597\" (UID: \"db0efa5b-3ced-44e4-ba5c-f3bdad377597\") " Nov 04 03:46:39.958840 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:46:39.958803 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db0efa5b-3ced-44e4-ba5c-f3bdad377597-kube-api-access-qmx56\" (UniqueName: \"kubernetes.io/projected/db0efa5b-3ced-44e4-ba5c-f3bdad377597-kube-api-access-qmx56\") pod \"db0efa5b-3ced-44e4-ba5c-f3bdad377597\" (UID: \"db0efa5b-3ced-44e4-ba5c-f3bdad377597\") " Nov 04 03:46:39.959033 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:46:39.959006 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db0efa5b-3ced-44e4-ba5c-f3bdad377597-openshift-service-ca" pod "db0efa5b-3ced-44e4-ba5c-f3bdad377597" (UID: "db0efa5b-3ced-44e4-ba5c-f3bdad377597"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:46:39.959213 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:46:39.959189 5953 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db0efa5b-3ced-44e4-ba5c-f3bdad377597-openshift-service-ca\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:46:39.960979 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:46:39.960951 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db0efa5b-3ced-44e4-ba5c-f3bdad377597-maas-api-tls" pod "db0efa5b-3ced-44e4-ba5c-f3bdad377597" (UID: "db0efa5b-3ced-44e4-ba5c-f3bdad377597"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:46:39.961082 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:46:39.960979 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db0efa5b-3ced-44e4-ba5c-f3bdad377597-metrics-certs" pod "db0efa5b-3ced-44e4-ba5c-f3bdad377597" (UID: "db0efa5b-3ced-44e4-ba5c-f3bdad377597"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:46:39.961167 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:46:39.961081 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db0efa5b-3ced-44e4-ba5c-f3bdad377597-kube-api-access-qmx56" pod "db0efa5b-3ced-44e4-ba5c-f3bdad377597" (UID: "db0efa5b-3ced-44e4-ba5c-f3bdad377597"). InnerVolumeSpecName "kube-api-access-qmx56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:46:40.060581 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:46:40.060548 5953 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db0efa5b-3ced-44e4-ba5c-f3bdad377597-maas-api-tls\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:46:40.060581 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:46:40.060578 5953 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db0efa5b-3ced-44e4-ba5c-f3bdad377597-metrics-certs\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:46:40.060581 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:46:40.060591 5953 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmx56\" (UniqueName: \"kubernetes.io/projected/db0efa5b-3ced-44e4-ba5c-f3bdad377597-kube-api-access-qmx56\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:46:40.104383 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:46:40.104350 5953 generic.go:356] "Generic (PLEG): container finished" podID="db0efa5b-3ced-44e4-ba5c-f3bdad377597" containerID="c6b09e9da904eaa5e56cf427947508aa0b4f03104393f458ef259e63af30ef36" exitCode=0 Nov 04 03:46:40.104543 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:46:40.104389 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g" event={"ID":"db0efa5b-3ced-44e4-ba5c-f3bdad377597","Type":"ContainerDied","Data":"c6b09e9da904eaa5e56cf427947508aa0b4f03104393f458ef259e63af30ef36"} Nov 04 03:46:40.104543 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:46:40.104440 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g" event={"ID":"db0efa5b-3ced-44e4-ba5c-f3bdad377597","Type":"ContainerDied","Data":"2fe3902d1300460854f74bd49272e5179e0efce12e5c01dd24a52b66b1702362"} Nov 04 03:46:40.104543 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:46:40.104463 5953 scope.go:122] "RemoveContainer" containerID="c6b09e9da904eaa5e56cf427947508aa0b4f03104393f458ef259e63af30ef36" Nov 04 03:46:40.104543 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:46:40.104402 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g" Nov 04 03:46:40.115504 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:46:40.115485 5953 scope.go:122] "RemoveContainer" containerID="c6b09e9da904eaa5e56cf427947508aa0b4f03104393f458ef259e63af30ef36" Nov 04 03:46:40.115821 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:46:40.115799 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6b09e9da904eaa5e56cf427947508aa0b4f03104393f458ef259e63af30ef36\": container with ID starting with c6b09e9da904eaa5e56cf427947508aa0b4f03104393f458ef259e63af30ef36 not found: ID does not exist" containerID="c6b09e9da904eaa5e56cf427947508aa0b4f03104393f458ef259e63af30ef36" Nov 04 03:46:40.115877 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:46:40.115831 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6b09e9da904eaa5e56cf427947508aa0b4f03104393f458ef259e63af30ef36"} err="failed to get container status \"c6b09e9da904eaa5e56cf427947508aa0b4f03104393f458ef259e63af30ef36\": rpc error: code = NotFound desc = could not find container \"c6b09e9da904eaa5e56cf427947508aa0b4f03104393f458ef259e63af30ef36\": container with ID starting with c6b09e9da904eaa5e56cf427947508aa0b4f03104393f458ef259e63af30ef36 not found: ID does not exist" Nov 04 03:46:40.270676 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:46:40.270633 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g"] Nov 04 03:46:40.276831 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:46:40.276806 5953 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-13190c-5cc67b6d6f-mn95g"] Nov 04 03:46:40.856896 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:46:40.856861 5953 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db0efa5b-3ced-44e4-ba5c-f3bdad377597" path="/var/lib/kubelet/pods/db0efa5b-3ced-44e4-ba5c-f3bdad377597/volumes" Nov 04 03:46:46.384105 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:46:46.384064 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:47:16.384513 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:47:16.384475 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:47:30.823961 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:30.823927 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27"] Nov 04 03:47:45.844702 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:45.844643 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27" podUID="40a71a57-c1dd-4119-83b9-22527ca0364f" containerName="main" containerID="cri-o://969533b92a44907f486f97cd11f223d10941daaefde41689cb61e2d43b53b398" gracePeriod=45 Nov 04 03:47:46.109390 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:46.109326 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27" Nov 04 03:47:46.169791 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:46.169755 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a71a57-c1dd-4119-83b9-22527ca0364f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40a71a57-c1dd-4119-83b9-22527ca0364f-tmp-dir\") pod \"40a71a57-c1dd-4119-83b9-22527ca0364f\" (UID: \"40a71a57-c1dd-4119-83b9-22527ca0364f\") " Nov 04 03:47:46.169791 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:46.169796 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a71a57-c1dd-4119-83b9-22527ca0364f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/40a71a57-c1dd-4119-83b9-22527ca0364f-dshm\") pod \"40a71a57-c1dd-4119-83b9-22527ca0364f\" (UID: \"40a71a57-c1dd-4119-83b9-22527ca0364f\") " Nov 04 03:47:46.170013 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:46.169832 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a71a57-c1dd-4119-83b9-22527ca0364f-home\" (UniqueName: \"kubernetes.io/empty-dir/40a71a57-c1dd-4119-83b9-22527ca0364f-home\") pod \"40a71a57-c1dd-4119-83b9-22527ca0364f\" (UID: \"40a71a57-c1dd-4119-83b9-22527ca0364f\") " Nov 04 03:47:46.170013 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:46.169916 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40a71a57-c1dd-4119-83b9-22527ca0364f-tls-certs\" (UniqueName: \"kubernetes.io/secret/40a71a57-c1dd-4119-83b9-22527ca0364f-tls-certs\") pod \"40a71a57-c1dd-4119-83b9-22527ca0364f\" (UID: \"40a71a57-c1dd-4119-83b9-22527ca0364f\") " Nov 04 03:47:46.170013 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:46.169972 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40a71a57-c1dd-4119-83b9-22527ca0364f-kube-api-access-2w2sb\" (UniqueName: \"kubernetes.io/projected/40a71a57-c1dd-4119-83b9-22527ca0364f-kube-api-access-2w2sb\") pod \"40a71a57-c1dd-4119-83b9-22527ca0364f\" (UID: \"40a71a57-c1dd-4119-83b9-22527ca0364f\") " Nov 04 03:47:46.170175 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:46.170021 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a71a57-c1dd-4119-83b9-22527ca0364f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40a71a57-c1dd-4119-83b9-22527ca0364f-model-cache\") pod \"40a71a57-c1dd-4119-83b9-22527ca0364f\" (UID: \"40a71a57-c1dd-4119-83b9-22527ca0364f\") " Nov 04 03:47:46.170175 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:46.170079 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a71a57-c1dd-4119-83b9-22527ca0364f-tmp-dir" pod "40a71a57-c1dd-4119-83b9-22527ca0364f" (UID: "40a71a57-c1dd-4119-83b9-22527ca0364f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:47:46.170284 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:46.170169 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a71a57-c1dd-4119-83b9-22527ca0364f-home" pod "40a71a57-c1dd-4119-83b9-22527ca0364f" (UID: "40a71a57-c1dd-4119-83b9-22527ca0364f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:47:46.170340 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:46.170305 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a71a57-c1dd-4119-83b9-22527ca0364f-model-cache" pod "40a71a57-c1dd-4119-83b9-22527ca0364f" (UID: "40a71a57-c1dd-4119-83b9-22527ca0364f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:47:46.170445 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:46.170424 5953 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40a71a57-c1dd-4119-83b9-22527ca0364f-model-cache\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:47:46.170513 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:46.170473 5953 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40a71a57-c1dd-4119-83b9-22527ca0364f-tmp-dir\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:47:46.170513 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:46.170488 5953 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40a71a57-c1dd-4119-83b9-22527ca0364f-home\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:47:46.172337 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:46.172309 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a71a57-c1dd-4119-83b9-22527ca0364f-dshm" pod "40a71a57-c1dd-4119-83b9-22527ca0364f" (UID: "40a71a57-c1dd-4119-83b9-22527ca0364f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:47:46.172460 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:46.172350 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40a71a57-c1dd-4119-83b9-22527ca0364f-tls-certs" pod "40a71a57-c1dd-4119-83b9-22527ca0364f" (UID: "40a71a57-c1dd-4119-83b9-22527ca0364f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:47:46.172503 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:46.172475 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40a71a57-c1dd-4119-83b9-22527ca0364f-kube-api-access-2w2sb" pod "40a71a57-c1dd-4119-83b9-22527ca0364f" (UID: "40a71a57-c1dd-4119-83b9-22527ca0364f"). InnerVolumeSpecName "kube-api-access-2w2sb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:47:46.271978 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:46.271947 5953 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40a71a57-c1dd-4119-83b9-22527ca0364f-tls-certs\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:47:46.271978 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:46.271978 5953 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w2sb\" (UniqueName: \"kubernetes.io/projected/40a71a57-c1dd-4119-83b9-22527ca0364f-kube-api-access-2w2sb\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:47:46.272182 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:46.271989 5953 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40a71a57-c1dd-4119-83b9-22527ca0364f-dshm\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:47:46.386846 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:47:46.386756 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:47:46.479787 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:46.479749 5953 generic.go:356] "Generic (PLEG): container finished" podID="40a71a57-c1dd-4119-83b9-22527ca0364f" containerID="969533b92a44907f486f97cd11f223d10941daaefde41689cb61e2d43b53b398" exitCode=0 Nov 04 03:47:46.479981 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:46.479799 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27" event={"ID":"40a71a57-c1dd-4119-83b9-22527ca0364f","Type":"ContainerDied","Data":"969533b92a44907f486f97cd11f223d10941daaefde41689cb61e2d43b53b398"} Nov 04 03:47:46.479981 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:46.479832 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27" event={"ID":"40a71a57-c1dd-4119-83b9-22527ca0364f","Type":"ContainerDied","Data":"4ee3b1effb924e20724aa27cf2bbc0b1b39d8ae3e0c1c0a9491efae27ae09990"} Nov 04 03:47:46.479981 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:46.479849 5953 scope.go:122] "RemoveContainer" containerID="969533b92a44907f486f97cd11f223d10941daaefde41689cb61e2d43b53b398" Nov 04 03:47:46.479981 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:46.479802 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27" Nov 04 03:47:46.491543 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:46.491520 5953 scope.go:122] "RemoveContainer" containerID="969533b92a44907f486f97cd11f223d10941daaefde41689cb61e2d43b53b398" Nov 04 03:47:46.491909 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:47:46.491886 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"969533b92a44907f486f97cd11f223d10941daaefde41689cb61e2d43b53b398\": container with ID starting with 969533b92a44907f486f97cd11f223d10941daaefde41689cb61e2d43b53b398 not found: ID does not exist" containerID="969533b92a44907f486f97cd11f223d10941daaefde41689cb61e2d43b53b398" Nov 04 03:47:46.491958 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:46.491920 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"969533b92a44907f486f97cd11f223d10941daaefde41689cb61e2d43b53b398"} err="failed to get container status \"969533b92a44907f486f97cd11f223d10941daaefde41689cb61e2d43b53b398\": rpc error: code = NotFound desc = could not find container \"969533b92a44907f486f97cd11f223d10941daaefde41689cb61e2d43b53b398\": container with ID starting with 969533b92a44907f486f97cd11f223d10941daaefde41689cb61e2d43b53b398 not found: ID does not exist" Nov 04 03:47:46.647357 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:46.647278 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27"] Nov 04 03:47:46.697100 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:46.697066 5953 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-c5bec0/distinct-w3-c5bec0-kserve-769854457b-mxs27"] Nov 04 03:47:46.857055 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:46.857023 5953 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40a71a57-c1dd-4119-83b9-22527ca0364f" path="/var/lib/kubelet/pods/40a71a57-c1dd-4119-83b9-22527ca0364f/volumes" Nov 04 03:47:51.773164 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:51.773128 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl"] Nov 04 03:47:51.773662 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:51.773413 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl" podUID="de73559f-5ec5-4db8-9329-289c3768ac60" containerName="maas-api" containerID="cri-o://fbdc963ae053144c5634b8155e11e7e7e3cbd6316b11756f8de72bf2aeb7caf7" gracePeriod=30 Nov 04 03:47:52.038525 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:52.038459 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl" Nov 04 03:47:52.133786 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:52.133748 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de73559f-5ec5-4db8-9329-289c3768ac60-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de73559f-5ec5-4db8-9329-289c3768ac60-maas-api-tls\") pod \"de73559f-5ec5-4db8-9329-289c3768ac60\" (UID: \"de73559f-5ec5-4db8-9329-289c3768ac60\") " Nov 04 03:47:52.133991 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:52.133840 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de73559f-5ec5-4db8-9329-289c3768ac60-metrics-certs\" (UniqueName: \"kubernetes.io/secret/de73559f-5ec5-4db8-9329-289c3768ac60-metrics-certs\") pod \"de73559f-5ec5-4db8-9329-289c3768ac60\" (UID: \"de73559f-5ec5-4db8-9329-289c3768ac60\") " Nov 04 03:47:52.133991 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:52.133928 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de73559f-5ec5-4db8-9329-289c3768ac60-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de73559f-5ec5-4db8-9329-289c3768ac60-openshift-service-ca\") pod \"de73559f-5ec5-4db8-9329-289c3768ac60\" (UID: \"de73559f-5ec5-4db8-9329-289c3768ac60\") " Nov 04 03:47:52.133991 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:52.133966 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de73559f-5ec5-4db8-9329-289c3768ac60-kube-api-access-8cqgx\" (UniqueName: \"kubernetes.io/projected/de73559f-5ec5-4db8-9329-289c3768ac60-kube-api-access-8cqgx\") pod \"de73559f-5ec5-4db8-9329-289c3768ac60\" (UID: \"de73559f-5ec5-4db8-9329-289c3768ac60\") " Nov 04 03:47:52.134274 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:52.134240 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de73559f-5ec5-4db8-9329-289c3768ac60-openshift-service-ca" pod "de73559f-5ec5-4db8-9329-289c3768ac60" (UID: "de73559f-5ec5-4db8-9329-289c3768ac60"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:47:52.134430 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:52.134399 5953 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de73559f-5ec5-4db8-9329-289c3768ac60-openshift-service-ca\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:47:52.136207 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:52.136182 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de73559f-5ec5-4db8-9329-289c3768ac60-metrics-certs" pod "de73559f-5ec5-4db8-9329-289c3768ac60" (UID: "de73559f-5ec5-4db8-9329-289c3768ac60"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:47:52.136327 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:52.136228 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de73559f-5ec5-4db8-9329-289c3768ac60-kube-api-access-8cqgx" pod "de73559f-5ec5-4db8-9329-289c3768ac60" (UID: "de73559f-5ec5-4db8-9329-289c3768ac60"). InnerVolumeSpecName "kube-api-access-8cqgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:47:52.136388 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:52.136329 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de73559f-5ec5-4db8-9329-289c3768ac60-maas-api-tls" pod "de73559f-5ec5-4db8-9329-289c3768ac60" (UID: "de73559f-5ec5-4db8-9329-289c3768ac60"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:47:52.235356 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:52.235324 5953 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de73559f-5ec5-4db8-9329-289c3768ac60-maas-api-tls\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:47:52.235356 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:52.235356 5953 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de73559f-5ec5-4db8-9329-289c3768ac60-metrics-certs\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:47:52.235599 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:52.235378 5953 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cqgx\" (UniqueName: \"kubernetes.io/projected/de73559f-5ec5-4db8-9329-289c3768ac60-kube-api-access-8cqgx\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:47:52.512232 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:52.512193 5953 generic.go:356] "Generic (PLEG): container finished" podID="de73559f-5ec5-4db8-9329-289c3768ac60" containerID="fbdc963ae053144c5634b8155e11e7e7e3cbd6316b11756f8de72bf2aeb7caf7" exitCode=0 Nov 04 03:47:52.512412 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:52.512247 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl" Nov 04 03:47:52.512412 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:52.512253 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl" event={"ID":"de73559f-5ec5-4db8-9329-289c3768ac60","Type":"ContainerDied","Data":"fbdc963ae053144c5634b8155e11e7e7e3cbd6316b11756f8de72bf2aeb7caf7"} Nov 04 03:47:52.512412 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:52.512305 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl" event={"ID":"de73559f-5ec5-4db8-9329-289c3768ac60","Type":"ContainerDied","Data":"0db16b1521369c7b3f9ca348368c2e6bd357e55ab2c02f03e63e7a3f2261bc94"} Nov 04 03:47:52.512412 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:52.512336 5953 scope.go:122] "RemoveContainer" containerID="fbdc963ae053144c5634b8155e11e7e7e3cbd6316b11756f8de72bf2aeb7caf7" Nov 04 03:47:52.522560 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:52.522538 5953 scope.go:122] "RemoveContainer" containerID="fbdc963ae053144c5634b8155e11e7e7e3cbd6316b11756f8de72bf2aeb7caf7" Nov 04 03:47:52.522930 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:47:52.522908 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbdc963ae053144c5634b8155e11e7e7e3cbd6316b11756f8de72bf2aeb7caf7\": container with ID starting with fbdc963ae053144c5634b8155e11e7e7e3cbd6316b11756f8de72bf2aeb7caf7 not found: ID does not exist" containerID="fbdc963ae053144c5634b8155e11e7e7e3cbd6316b11756f8de72bf2aeb7caf7" Nov 04 03:47:52.522993 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:52.522938 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbdc963ae053144c5634b8155e11e7e7e3cbd6316b11756f8de72bf2aeb7caf7"} err="failed to get container status \"fbdc963ae053144c5634b8155e11e7e7e3cbd6316b11756f8de72bf2aeb7caf7\": rpc error: code = NotFound desc = could not find container \"fbdc963ae053144c5634b8155e11e7e7e3cbd6316b11756f8de72bf2aeb7caf7\": container with ID starting with fbdc963ae053144c5634b8155e11e7e7e3cbd6316b11756f8de72bf2aeb7caf7 not found: ID does not exist" Nov 04 03:47:52.690321 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:52.690285 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl"] Nov 04 03:47:52.694840 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:52.694817 5953 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-c5bec0-6d89b6b97d-qn9rl"] Nov 04 03:47:52.855012 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:47:52.854936 5953 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de73559f-5ec5-4db8-9329-289c3768ac60" path="/var/lib/kubelet/pods/de73559f-5ec5-4db8-9329-289c3768ac60/volumes" Nov 04 03:48:12.035558 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:12.035475 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f"] Nov 04 03:48:12.036071 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:12.035739 5953 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" podUID="cdfcbe1f-229d-4c24-9e85-01e59dac657a" containerName="istio-proxy" containerID="cri-o://fff3383cc2cb66f3d65434a859143631e9106ed7b51ce4bf62741ef0826c0a52" gracePeriod=30 Nov 04 03:48:13.227454 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:13.227413 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-874888cd5-l4bhw_a67b383f-8c42-4aca-a45c-2666a066a80e/authorino/0.log" Nov 04 03:48:13.613215 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:13.613174 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-xzvqn_eef0a147-2a4f-48fe-bc09-c96b8d05e888/manager/0.log" Nov 04 03:48:15.787179 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:15.787152 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-dcl2t_0c122db7-3d8a-4aec-b046-8e02811c7082/manager/0.log" Nov 04 03:48:16.385517 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:48:16.385479 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:48:16.928163 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:16.928133 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-d69f556c7-qx7mw_60f0eec5-a469-42ad-8f4f-d11a9b840081/manager/0.log" Nov 04 03:48:17.295213 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.295189 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" Nov 04 03:48:17.388876 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.388841 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdfcbe1f-229d-4c24-9e85-01e59dac657a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdfcbe1f-229d-4c24-9e85-01e59dac657a-workload-socket\") pod \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\" (UID: \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\") " Nov 04 03:48:17.389070 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.388935 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdfcbe1f-229d-4c24-9e85-01e59dac657a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdfcbe1f-229d-4c24-9e85-01e59dac657a-istio-data\") pod \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\" (UID: \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\") " Nov 04 03:48:17.389070 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.388979 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdfcbe1f-229d-4c24-9e85-01e59dac657a-istio-token\" (UniqueName: \"kubernetes.io/projected/cdfcbe1f-229d-4c24-9e85-01e59dac657a-istio-token\") pod \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\" (UID: \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\") " Nov 04 03:48:17.389070 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.389004 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdfcbe1f-229d-4c24-9e85-01e59dac657a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdfcbe1f-229d-4c24-9e85-01e59dac657a-istiod-ca-cert\") pod \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\" (UID: \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\") " Nov 04 03:48:17.389070 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.389029 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdfcbe1f-229d-4c24-9e85-01e59dac657a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdfcbe1f-229d-4c24-9e85-01e59dac657a-workload-certs\") pod \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\" (UID: \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\") " Nov 04 03:48:17.389070 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.389061 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdfcbe1f-229d-4c24-9e85-01e59dac657a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdfcbe1f-229d-4c24-9e85-01e59dac657a-credential-socket\") pod \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\" (UID: \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\") " Nov 04 03:48:17.389331 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.389088 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdfcbe1f-229d-4c24-9e85-01e59dac657a-kube-api-access-dgb8x\" (UniqueName: \"kubernetes.io/projected/cdfcbe1f-229d-4c24-9e85-01e59dac657a-kube-api-access-dgb8x\") pod \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\" (UID: \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\") " Nov 04 03:48:17.389331 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.389136 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdfcbe1f-229d-4c24-9e85-01e59dac657a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdfcbe1f-229d-4c24-9e85-01e59dac657a-istio-envoy\") pod \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\" (UID: \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\") " Nov 04 03:48:17.389331 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.389162 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdfcbe1f-229d-4c24-9e85-01e59dac657a-workload-socket" pod "cdfcbe1f-229d-4c24-9e85-01e59dac657a" (UID: "cdfcbe1f-229d-4c24-9e85-01e59dac657a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:48:17.389331 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.389260 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdfcbe1f-229d-4c24-9e85-01e59dac657a-istio-data" pod "cdfcbe1f-229d-4c24-9e85-01e59dac657a" (UID: "cdfcbe1f-229d-4c24-9e85-01e59dac657a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:48:17.389331 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.389293 5953 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cdfcbe1f-229d-4c24-9e85-01e59dac657a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdfcbe1f-229d-4c24-9e85-01e59dac657a-istio-podinfo\") pod \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\" (UID: \"cdfcbe1f-229d-4c24-9e85-01e59dac657a\") " Nov 04 03:48:17.389585 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.389381 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdfcbe1f-229d-4c24-9e85-01e59dac657a-workload-certs" pod "cdfcbe1f-229d-4c24-9e85-01e59dac657a" (UID: "cdfcbe1f-229d-4c24-9e85-01e59dac657a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:48:17.389585 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.389468 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdfcbe1f-229d-4c24-9e85-01e59dac657a-istiod-ca-cert" pod "cdfcbe1f-229d-4c24-9e85-01e59dac657a" (UID: "cdfcbe1f-229d-4c24-9e85-01e59dac657a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:48:17.389585 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.389485 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdfcbe1f-229d-4c24-9e85-01e59dac657a-credential-socket" pod "cdfcbe1f-229d-4c24-9e85-01e59dac657a" (UID: "cdfcbe1f-229d-4c24-9e85-01e59dac657a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:48:17.389772 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.389749 5953 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdfcbe1f-229d-4c24-9e85-01e59dac657a-workload-socket\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:48:17.389927 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.389846 5953 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdfcbe1f-229d-4c24-9e85-01e59dac657a-istio-data\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:48:17.390197 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.389953 5953 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdfcbe1f-229d-4c24-9e85-01e59dac657a-istiod-ca-cert\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:48:17.390197 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.389971 5953 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdfcbe1f-229d-4c24-9e85-01e59dac657a-workload-certs\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:48:17.390197 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.389985 5953 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdfcbe1f-229d-4c24-9e85-01e59dac657a-credential-socket\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:48:17.391504 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.391474 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdfcbe1f-229d-4c24-9e85-01e59dac657a-istio-token" pod "cdfcbe1f-229d-4c24-9e85-01e59dac657a" (UID: "cdfcbe1f-229d-4c24-9e85-01e59dac657a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:48:17.391609 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.391584 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdfcbe1f-229d-4c24-9e85-01e59dac657a-istio-envoy" pod "cdfcbe1f-229d-4c24-9e85-01e59dac657a" (UID: "cdfcbe1f-229d-4c24-9e85-01e59dac657a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:48:17.391692 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.391661 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cdfcbe1f-229d-4c24-9e85-01e59dac657a-istio-podinfo" pod "cdfcbe1f-229d-4c24-9e85-01e59dac657a" (UID: "cdfcbe1f-229d-4c24-9e85-01e59dac657a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:48:17.391692 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.391666 5953 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdfcbe1f-229d-4c24-9e85-01e59dac657a-kube-api-access-dgb8x" pod "cdfcbe1f-229d-4c24-9e85-01e59dac657a" (UID: "cdfcbe1f-229d-4c24-9e85-01e59dac657a"). InnerVolumeSpecName "kube-api-access-dgb8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:48:17.463205 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.463128 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-f6dp2_2bc38436-c34c-4561-9463-07b5edc984e7/manager/0.log" Nov 04 03:48:17.490604 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.490568 5953 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdfcbe1f-229d-4c24-9e85-01e59dac657a-istio-podinfo\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:48:17.490604 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.490598 5953 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdfcbe1f-229d-4c24-9e85-01e59dac657a-istio-token\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:48:17.490604 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.490611 5953 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgb8x\" (UniqueName: \"kubernetes.io/projected/cdfcbe1f-229d-4c24-9e85-01e59dac657a-kube-api-access-dgb8x\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:48:17.490862 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.490638 5953 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdfcbe1f-229d-4c24-9e85-01e59dac657a-istio-envoy\") on node \"ip-10-0-133-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:48:17.640500 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.640462 5953 generic.go:356] "Generic (PLEG): container finished" podID="cdfcbe1f-229d-4c24-9e85-01e59dac657a" containerID="fff3383cc2cb66f3d65434a859143631e9106ed7b51ce4bf62741ef0826c0a52" exitCode=0 Nov 04 03:48:17.640724 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.640522 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" event={"ID":"cdfcbe1f-229d-4c24-9e85-01e59dac657a","Type":"ContainerDied","Data":"fff3383cc2cb66f3d65434a859143631e9106ed7b51ce4bf62741ef0826c0a52"} Nov 04 03:48:17.640724 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.640526 5953 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" Nov 04 03:48:17.640724 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.640589 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f" event={"ID":"cdfcbe1f-229d-4c24-9e85-01e59dac657a","Type":"ContainerDied","Data":"dbcb32d2fe164b6bd5087b67fef62276af624ae709e920bf1671ff2a8118e64b"} Nov 04 03:48:17.640724 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.640610 5953 scope.go:122] "RemoveContainer" containerID="fff3383cc2cb66f3d65434a859143631e9106ed7b51ce4bf62741ef0826c0a52" Nov 04 03:48:17.650808 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.650715 5953 scope.go:122] "RemoveContainer" containerID="15061a35835d7a806d91781392f465eb88d2d8dad663d6687aa49a7db223b4b7" Nov 04 03:48:17.660103 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.660075 5953 scope.go:122] "RemoveContainer" containerID="fff3383cc2cb66f3d65434a859143631e9106ed7b51ce4bf62741ef0826c0a52" Nov 04 03:48:17.660429 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:48:17.660405 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fff3383cc2cb66f3d65434a859143631e9106ed7b51ce4bf62741ef0826c0a52\": container with ID starting with fff3383cc2cb66f3d65434a859143631e9106ed7b51ce4bf62741ef0826c0a52 not found: ID does not exist" containerID="fff3383cc2cb66f3d65434a859143631e9106ed7b51ce4bf62741ef0826c0a52" Nov 04 03:48:17.660526 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.660437 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fff3383cc2cb66f3d65434a859143631e9106ed7b51ce4bf62741ef0826c0a52"} err="failed to get container status \"fff3383cc2cb66f3d65434a859143631e9106ed7b51ce4bf62741ef0826c0a52\": rpc error: code = NotFound desc = could not find container \"fff3383cc2cb66f3d65434a859143631e9106ed7b51ce4bf62741ef0826c0a52\": container with ID starting with fff3383cc2cb66f3d65434a859143631e9106ed7b51ce4bf62741ef0826c0a52 not found: ID does not exist" Nov 04 03:48:17.660526 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.660457 5953 scope.go:122] "RemoveContainer" containerID="15061a35835d7a806d91781392f465eb88d2d8dad663d6687aa49a7db223b4b7" Nov 04 03:48:17.660766 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:48:17.660747 5953 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15061a35835d7a806d91781392f465eb88d2d8dad663d6687aa49a7db223b4b7\": container with ID starting with 15061a35835d7a806d91781392f465eb88d2d8dad663d6687aa49a7db223b4b7 not found: ID does not exist" containerID="15061a35835d7a806d91781392f465eb88d2d8dad663d6687aa49a7db223b4b7" Nov 04 03:48:17.660829 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.660774 5953 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15061a35835d7a806d91781392f465eb88d2d8dad663d6687aa49a7db223b4b7"} err="failed to get container status \"15061a35835d7a806d91781392f465eb88d2d8dad663d6687aa49a7db223b4b7\": rpc error: code = NotFound desc = could not find container \"15061a35835d7a806d91781392f465eb88d2d8dad663d6687aa49a7db223b4b7\": container with ID starting with 15061a35835d7a806d91781392f465eb88d2d8dad663d6687aa49a7db223b4b7 not found: ID does not exist" Nov 04 03:48:17.806807 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.806767 5953 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f"] Nov 04 03:48:17.812396 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:17.812364 5953 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-ltg2f"] Nov 04 03:48:18.202570 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:18.202490 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-868857dbd9-6b89z_60409783-fdb7-4b2e-9ce3-05fd3be2ab8c/manager/0.log" Nov 04 03:48:18.474192 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:18.474111 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-w5ktz_94939ede-2ec5-4439-b25b-30e7cf09a65e/server/0.log" Nov 04 03:48:18.854912 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:18.854881 5953 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdfcbe1f-229d-4c24-9e85-01e59dac657a" path="/var/lib/kubelet/pods/cdfcbe1f-229d-4c24-9e85-01e59dac657a/volumes" Nov 04 03:48:18.962314 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:18.962282 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-tt4m7_e53b6123-3a57-4a27-87e4-d430f8d8e4cc/proxy/0.log" Nov 04 03:48:23.125772 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:23.125741 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-874888cd5-l4bhw_a67b383f-8c42-4aca-a45c-2666a066a80e/authorino/0.log" Nov 04 03:48:46.384751 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:48:46.384716 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:48:54.802296 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:54.802262 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-d69f556c7-qx7mw_60f0eec5-a469-42ad-8f4f-d11a9b840081/manager/0.log" Nov 04 03:48:55.369414 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:55.369384 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-f6dp2_2bc38436-c34c-4561-9463-07b5edc984e7/manager/0.log" Nov 04 03:48:56.101554 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:56.101527 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-868857dbd9-6b89z_60409783-fdb7-4b2e-9ce3-05fd3be2ab8c/manager/0.log" Nov 04 03:48:56.545526 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:56.545497 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-w5ktz_94939ede-2ec5-4439-b25b-30e7cf09a65e/server/0.log" Nov 04 03:48:57.024922 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:48:57.024889 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-tt4m7_e53b6123-3a57-4a27-87e4-d430f8d8e4cc/proxy/0.log" Nov 04 03:49:11.103862 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:49:11.103835 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-874888cd5-l4bhw_a67b383f-8c42-4aca-a45c-2666a066a80e/authorino/0.log" Nov 04 03:49:11.497607 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:49:11.497531 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-xzvqn_eef0a147-2a4f-48fe-bc09-c96b8d05e888/manager/0.log" Nov 04 03:49:13.312955 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:49:13.312926 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-dcl2t_0c122db7-3d8a-4aec-b046-8e02811c7082/manager/0.log" Nov 04 03:49:15.044421 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:49:15.044385 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d77786fb9-9gmhr_b7f96f38-4540-4d54-a792-35996a8767c8/kube-auth-proxy/0.log" Nov 04 03:49:15.500730 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:49:15.500702 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-2nvtc_75dde092-12d7-4855-a6ba-3edb9e650a5c/istio-proxy/0.log" Nov 04 03:49:16.385540 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:49:16.385497 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:49:17.247368 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:49:17.247338 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-rmgs4_c16e30a6-2c91-4687-91ff-6b496c753277/istio-proxy/0.log" Nov 04 03:49:17.467792 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:49:17.467759 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-swnrt_de0a15d7-f04e-49e1-a31b-1f46cfa50333/istio-proxy/0.log" Nov 04 03:49:18.176370 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:49:18.176339 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-86fmg_bab1308d-3935-4601-acb9-48ac9ba1cdc9/payload-pre-processing/0.log" Nov 04 03:49:18.409024 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:49:18.408991 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-vzvct_050f4e90-30e5-4f04-9293-63424357799f/payload-processing/0.log" Nov 04 03:49:29.746247 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:49:29.746220 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-zs2mg_83e293dd-236c-4b48-a841-27dd9190db27/main/0.log" Nov 04 03:49:29.970134 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:49:29.970103 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-cmqzr_51318cd3-19e0-40ca-8731-78f8b0d18a26/main/0.log" Nov 04 03:49:30.225350 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:49:30.225324 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-ftccw_5a08f9b6-140f-4f6b-974a-dcce183d79c8/main/0.log" Nov 04 03:49:30.716974 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:49:30.716939 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x4bcq_dc86bf58-18cd-4838-a56b-7a88dde20533/main/0.log" Nov 04 03:49:31.226735 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:49:31.226708 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-4kx7c_3d1425e2-49d3-462b-b7b5-85fd5e7adaf4/main/0.log" Nov 04 03:49:46.388254 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:49:46.388212 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:50:01.074506 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:01.074474 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8rmjv_834b5587-1820-4089-a925-ab992fe654b4/global-pull-secret-syncer/0.log" Nov 04 03:50:05.051585 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:05.051557 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-d69f556c7-qx7mw_60f0eec5-a469-42ad-8f4f-d11a9b840081/manager/0.log" Nov 04 03:50:05.242582 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:05.242554 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pmv6n_923f8598-9df4-46bb-8da0-73442c97522f/konnectivity-agent/0.log" Nov 04 03:50:05.511352 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:05.511325 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-f6dp2_2bc38436-c34c-4561-9463-07b5edc984e7/manager/0.log" Nov 04 03:50:06.052746 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:06.052718 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-868857dbd9-6b89z_60409783-fdb7-4b2e-9ce3-05fd3be2ab8c/manager/0.log" Nov 04 03:50:06.216065 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:06.216039 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-w5ktz_94939ede-2ec5-4439-b25b-30e7cf09a65e/server/0.log" Nov 04 03:50:06.597086 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:06.597055 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-6.us-west-1.compute.internal_ca990c51a9c4be7e8fb143b232789941/haproxy/0.log" Nov 04 03:50:07.262592 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:07.262568 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-tt4m7_e53b6123-3a57-4a27-87e4-d430f8d8e4cc/proxy/0.log" Nov 04 03:50:10.655143 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:10.655112 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-d69f556c7-qx7mw_60f0eec5-a469-42ad-8f4f-d11a9b840081/manager/0.log" Nov 04 03:50:11.289245 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:11.289214 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-874888cd5-l4bhw_a67b383f-8c42-4aca-a45c-2666a066a80e/authorino/0.log" Nov 04 03:50:11.340804 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:11.340775 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-f6dp2_2bc38436-c34c-4561-9463-07b5edc984e7/manager/0.log" Nov 04 03:50:11.613495 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:11.613399 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-xzvqn_eef0a147-2a4f-48fe-bc09-c96b8d05e888/manager/0.log" Nov 04 03:50:11.870476 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:11.870387 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-868857dbd9-6b89z_60409783-fdb7-4b2e-9ce3-05fd3be2ab8c/manager/0.log" Nov 04 03:50:12.032140 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.032112 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-w5ktz_94939ede-2ec5-4439-b25b-30e7cf09a65e/server/0.log" Nov 04 03:50:12.626490 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.626454 5953 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rnkxc/perf-node-gather-daemonset-xpwfp"] Nov 04 03:50:12.626974 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.626960 5953 cpu_manager.go:414] "containerMap: removing container" podUID="db0efa5b-3ced-44e4-ba5c-f3bdad377597" containerName="maas-api" Nov 04 03:50:12.627028 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.626975 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db0efa5b-3ced-44e4-ba5c-f3bdad377597" containerName="maas-api" Nov 04 03:50:12.627028 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.626988 5953 cpu_manager.go:414] "containerMap: removing container" podUID="020ca3c3-bc40-4131-8e81-6f4bb1ffefdb" containerName="cleanup" Nov 04 03:50:12.627028 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.626993 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="020ca3c3-bc40-4131-8e81-6f4bb1ffefdb" containerName="cleanup" Nov 04 03:50:12.627028 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.627022 5953 cpu_manager.go:414] "containerMap: removing container" podUID="40a71a57-c1dd-4119-83b9-22527ca0364f" containerName="main" Nov 04 03:50:12.627028 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.627027 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40a71a57-c1dd-4119-83b9-22527ca0364f" containerName="main" Nov 04 03:50:12.627178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.627037 5953 cpu_manager.go:414] "containerMap: removing container" podUID="020ca3c3-bc40-4131-8e81-6f4bb1ffefdb" containerName="cleanup" Nov 04 03:50:12.627178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.627042 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="020ca3c3-bc40-4131-8e81-6f4bb1ffefdb" containerName="cleanup" Nov 04 03:50:12.627178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.627055 5953 cpu_manager.go:414] "containerMap: removing container" podUID="de73559f-5ec5-4db8-9329-289c3768ac60" containerName="maas-api" Nov 04 03:50:12.627178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.627060 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de73559f-5ec5-4db8-9329-289c3768ac60" containerName="maas-api" Nov 04 03:50:12.627178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.627070 5953 cpu_manager.go:414] "containerMap: removing container" podUID="cdfcbe1f-229d-4c24-9e85-01e59dac657a" containerName="istio-proxy" Nov 04 03:50:12.627178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.627076 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdfcbe1f-229d-4c24-9e85-01e59dac657a" containerName="istio-proxy" Nov 04 03:50:12.627178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.627084 5953 cpu_manager.go:414] "containerMap: removing container" podUID="cdfcbe1f-229d-4c24-9e85-01e59dac657a" containerName="istio-proxy" Nov 04 03:50:12.627178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.627089 5953 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdfcbe1f-229d-4c24-9e85-01e59dac657a" containerName="istio-proxy" Nov 04 03:50:12.627178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.627148 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="40a71a57-c1dd-4119-83b9-22527ca0364f" containerName="main" Nov 04 03:50:12.627178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.627158 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="020ca3c3-bc40-4131-8e81-6f4bb1ffefdb" containerName="cleanup" Nov 04 03:50:12.627178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.627165 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="de73559f-5ec5-4db8-9329-289c3768ac60" containerName="maas-api" Nov 04 03:50:12.627178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.627172 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdfcbe1f-229d-4c24-9e85-01e59dac657a" containerName="istio-proxy" Nov 04 03:50:12.627178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.627178 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="020ca3c3-bc40-4131-8e81-6f4bb1ffefdb" containerName="cleanup" Nov 04 03:50:12.627178 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.627183 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdfcbe1f-229d-4c24-9e85-01e59dac657a" containerName="istio-proxy" Nov 04 03:50:12.627579 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.627189 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="db0efa5b-3ced-44e4-ba5c-f3bdad377597" containerName="maas-api" Nov 04 03:50:12.627579 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.627195 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="020ca3c3-bc40-4131-8e81-6f4bb1ffefdb" containerName="cleanup" Nov 04 03:50:12.627579 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.627201 5953 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9d98901-eb52-4bd0-a031-640623bd4272" containerName="istio-proxy" Nov 04 03:50:12.630332 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.630318 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-xpwfp" Nov 04 03:50:12.686250 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.686213 5953 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rnkxc\"/\"default-dockercfg-hbjsz\"" Nov 04 03:50:12.736486 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.736453 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rnkxc\"/\"openshift-service-ca.crt\"" Nov 04 03:50:12.736486 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.736453 5953 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rnkxc\"/\"kube-root-ca.crt\"" Nov 04 03:50:12.738400 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.738372 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rnkxc/perf-node-gather-daemonset-xpwfp"] Nov 04 03:50:12.767902 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.767871 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c12910c0-a48e-4a4a-9cd7-111e0beb260a-podres\") pod \"perf-node-gather-daemonset-xpwfp\" (UID: \"c12910c0-a48e-4a4a-9cd7-111e0beb260a\") " pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-xpwfp" Nov 04 03:50:12.768066 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.768022 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c12910c0-a48e-4a4a-9cd7-111e0beb260a-sys\") pod \"perf-node-gather-daemonset-xpwfp\" (UID: \"c12910c0-a48e-4a4a-9cd7-111e0beb260a\") " pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-xpwfp" Nov 04 03:50:12.768141 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.768127 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7jfl\" (UniqueName: \"kubernetes.io/projected/c12910c0-a48e-4a4a-9cd7-111e0beb260a-kube-api-access-l7jfl\") pod \"perf-node-gather-daemonset-xpwfp\" (UID: \"c12910c0-a48e-4a4a-9cd7-111e0beb260a\") " pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-xpwfp" Nov 04 03:50:12.768180 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.768152 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c12910c0-a48e-4a4a-9cd7-111e0beb260a-lib-modules\") pod \"perf-node-gather-daemonset-xpwfp\" (UID: \"c12910c0-a48e-4a4a-9cd7-111e0beb260a\") " pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-xpwfp" Nov 04 03:50:12.768221 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.768186 5953 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c12910c0-a48e-4a4a-9cd7-111e0beb260a-proc\") pod \"perf-node-gather-daemonset-xpwfp\" (UID: \"c12910c0-a48e-4a4a-9cd7-111e0beb260a\") " pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-xpwfp" Nov 04 03:50:12.869359 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.869330 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c12910c0-a48e-4a4a-9cd7-111e0beb260a-sys\") pod \"perf-node-gather-daemonset-xpwfp\" (UID: \"c12910c0-a48e-4a4a-9cd7-111e0beb260a\") " pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-xpwfp" Nov 04 03:50:12.869532 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.869391 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7jfl\" (UniqueName: \"kubernetes.io/projected/c12910c0-a48e-4a4a-9cd7-111e0beb260a-kube-api-access-l7jfl\") pod \"perf-node-gather-daemonset-xpwfp\" (UID: \"c12910c0-a48e-4a4a-9cd7-111e0beb260a\") " pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-xpwfp" Nov 04 03:50:12.869532 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.869423 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c12910c0-a48e-4a4a-9cd7-111e0beb260a-lib-modules\") pod \"perf-node-gather-daemonset-xpwfp\" (UID: \"c12910c0-a48e-4a4a-9cd7-111e0beb260a\") " pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-xpwfp" Nov 04 03:50:12.869532 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.869448 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c12910c0-a48e-4a4a-9cd7-111e0beb260a-sys\") pod \"perf-node-gather-daemonset-xpwfp\" (UID: \"c12910c0-a48e-4a4a-9cd7-111e0beb260a\") " pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-xpwfp" Nov 04 03:50:12.869532 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.869475 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c12910c0-a48e-4a4a-9cd7-111e0beb260a-proc\") pod \"perf-node-gather-daemonset-xpwfp\" (UID: \"c12910c0-a48e-4a4a-9cd7-111e0beb260a\") " pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-xpwfp" Nov 04 03:50:12.869532 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.869529 5953 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c12910c0-a48e-4a4a-9cd7-111e0beb260a-podres\") pod \"perf-node-gather-daemonset-xpwfp\" (UID: \"c12910c0-a48e-4a4a-9cd7-111e0beb260a\") " pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-xpwfp" Nov 04 03:50:12.869724 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.869555 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c12910c0-a48e-4a4a-9cd7-111e0beb260a-lib-modules\") pod \"perf-node-gather-daemonset-xpwfp\" (UID: \"c12910c0-a48e-4a4a-9cd7-111e0beb260a\") " pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-xpwfp" Nov 04 03:50:12.869724 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.869576 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c12910c0-a48e-4a4a-9cd7-111e0beb260a-proc\") pod \"perf-node-gather-daemonset-xpwfp\" (UID: \"c12910c0-a48e-4a4a-9cd7-111e0beb260a\") " pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-xpwfp" Nov 04 03:50:12.869724 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.869614 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c12910c0-a48e-4a4a-9cd7-111e0beb260a-podres\") pod \"perf-node-gather-daemonset-xpwfp\" (UID: \"c12910c0-a48e-4a4a-9cd7-111e0beb260a\") " pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-xpwfp" Nov 04 03:50:12.931351 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.931269 5953 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7jfl\" (UniqueName: \"kubernetes.io/projected/c12910c0-a48e-4a4a-9cd7-111e0beb260a-kube-api-access-l7jfl\") pod \"perf-node-gather-daemonset-xpwfp\" (UID: \"c12910c0-a48e-4a4a-9cd7-111e0beb260a\") " pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-xpwfp" Nov 04 03:50:12.943269 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:12.943238 5953 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-xpwfp" Nov 04 03:50:13.141894 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:13.141855 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-tt4m7_e53b6123-3a57-4a27-87e4-d430f8d8e4cc/proxy/0.log" Nov 04 03:50:13.327609 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:13.327582 5953 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rnkxc/perf-node-gather-daemonset-xpwfp"] Nov 04 03:50:13.330760 ip-10-0-133-6 kubenswrapper[5953]: W1104 03:50:13.330729 5953 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc12910c0_a48e_4a4a_9cd7_111e0beb260a.slice/crio-98f4bda8379dfd1cc4da4235450e24a5d5b818c590059b388fb45e172f51e1d2 WatchSource:0}: Error finding container 98f4bda8379dfd1cc4da4235450e24a5d5b818c590059b388fb45e172f51e1d2: Status 404 returned error can't find the container with id 98f4bda8379dfd1cc4da4235450e24a5d5b818c590059b388fb45e172f51e1d2 Nov 04 03:50:14.052099 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:14.052067 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-dcl2t_0c122db7-3d8a-4aec-b046-8e02811c7082/manager/0.log" Nov 04 03:50:14.239301 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:14.239264 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-xpwfp" event={"ID":"c12910c0-a48e-4a4a-9cd7-111e0beb260a","Type":"ContainerStarted","Data":"8463ef84333d63770468862a168c4f155abe283ef1a731cf13a74bb28d895086"} Nov 04 03:50:14.239521 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:14.239369 5953 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-xpwfp" event={"ID":"c12910c0-a48e-4a4a-9cd7-111e0beb260a","Type":"ContainerStarted","Data":"98f4bda8379dfd1cc4da4235450e24a5d5b818c590059b388fb45e172f51e1d2"} Nov 04 03:50:14.239521 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:14.239408 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-xpwfp" Nov 04 03:50:14.356694 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:14.356577 5953 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-xpwfp" podStartSLOduration=2.356562685 podStartE2EDuration="2.356562685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 03:50:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:50:14.355392242 +0000 UTC m=+1939.630022998" watchObservedRunningTime="2026-11-04 03:50:14.356562685 +0000 UTC m=+1939.631193447" Nov 04 03:50:16.180239 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:16.180200 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-d69f556c7-qx7mw_60f0eec5-a469-42ad-8f4f-d11a9b840081/manager/0.log" Nov 04 03:50:16.386408 ip-10-0-133-6 kubenswrapper[5953]: E1104 03:50:16.386370 5953 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p8x4k" containerName="extractor" Nov 04 03:50:16.615339 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:16.615308 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-f6dp2_2bc38436-c34c-4561-9463-07b5edc984e7/manager/0.log" Nov 04 03:50:16.662903 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:16.662878 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c124d431-5af1-4978-9a20-cd4637befe01/alertmanager/0.log" Nov 04 03:50:17.081782 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:17.081740 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c124d431-5af1-4978-9a20-cd4637befe01/config-reloader/0.log" Nov 04 03:50:17.135701 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:17.135668 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-868857dbd9-6b89z_60409783-fdb7-4b2e-9ce3-05fd3be2ab8c/manager/0.log" Nov 04 03:50:17.309823 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:17.309786 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-w5ktz_94939ede-2ec5-4439-b25b-30e7cf09a65e/server/0.log" Nov 04 03:50:17.512167 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:17.512136 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c124d431-5af1-4978-9a20-cd4637befe01/kube-rbac-proxy-web/0.log" Nov 04 03:50:17.930158 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:17.930077 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c124d431-5af1-4978-9a20-cd4637befe01/kube-rbac-proxy/0.log" Nov 04 03:50:18.349398 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:18.349368 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c124d431-5af1-4978-9a20-cd4637befe01/kube-rbac-proxy-metric/0.log" Nov 04 03:50:18.377475 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:18.377436 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-tt4m7_e53b6123-3a57-4a27-87e4-d430f8d8e4cc/proxy/0.log" Nov 04 03:50:18.770035 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:18.769994 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c124d431-5af1-4978-9a20-cd4637befe01/prom-label-proxy/0.log" Nov 04 03:50:19.189999 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:19.189921 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c124d431-5af1-4978-9a20-cd4637befe01/init-config-reloader/0.log" Nov 04 03:50:20.054090 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:20.054055 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-ztrlc_57272665-8b2d-4854-bcb6-98f662716f4c/kube-state-metrics/0.log" Nov 04 03:50:20.255354 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:20.255330 5953 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-xpwfp" Nov 04 03:50:20.469128 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:20.469039 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-ztrlc_57272665-8b2d-4854-bcb6-98f662716f4c/kube-rbac-proxy-main/0.log" Nov 04 03:50:20.887100 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:20.887058 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-ztrlc_57272665-8b2d-4854-bcb6-98f662716f4c/kube-rbac-proxy-self/0.log" Nov 04 03:50:21.309681 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:21.309649 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-748ff79954-pr5lc_bbf7bd2f-e4c8-469f-ad1b-ffe5cf1fc271/metrics-server/0.log" Nov 04 03:50:21.730952 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:21.730879 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-kf7dm_cd03c8ce-341a-4afe-93a1-2280392dd4f8/monitoring-plugin/0.log" Nov 04 03:50:22.331822 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:22.331792 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7c9tl_16659cac-4c32-4a94-90aa-8be0c6be33b2/node-exporter/0.log" Nov 04 03:50:22.779727 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:22.779703 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7c9tl_16659cac-4c32-4a94-90aa-8be0c6be33b2/kube-rbac-proxy/0.log" Nov 04 03:50:23.198857 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:23.198781 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7c9tl_16659cac-4c32-4a94-90aa-8be0c6be33b2/init-textfile/0.log" Nov 04 03:50:23.481092 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:23.481007 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d77786fb9-9gmhr_b7f96f38-4540-4d54-a792-35996a8767c8/kube-auth-proxy/0.log" Nov 04 03:50:23.919636 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:23.919598 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-2nvtc_75dde092-12d7-4855-a6ba-3edb9e650a5c/istio-proxy/0.log" Nov 04 03:50:25.473254 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:25.473220 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-rmgs4_c16e30a6-2c91-4687-91ff-6b496c753277/istio-proxy/0.log" Nov 04 03:50:25.688575 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:25.688545 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-swnrt_de0a15d7-f04e-49e1-a31b-1f46cfa50333/istio-proxy/0.log" Nov 04 03:50:26.171405 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:26.171367 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-jf99x_a040e5d8-7316-4a37-a045-9eb7c2eaf607/kube-rbac-proxy-main/0.log" Nov 04 03:50:26.336594 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:26.336569 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-86fmg_bab1308d-3935-4601-acb9-48ac9ba1cdc9/payload-pre-processing/0.log" Nov 04 03:50:26.549334 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:26.549303 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-vzvct_050f4e90-30e5-4f04-9293-63424357799f/payload-processing/0.log" Nov 04 03:50:26.590682 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:26.590650 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-jf99x_a040e5d8-7316-4a37-a045-9eb7c2eaf607/kube-rbac-proxy-self/0.log" Nov 04 03:50:27.009283 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:27.009248 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-jf99x_a040e5d8-7316-4a37-a045-9eb7c2eaf607/openshift-state-metrics/0.log" Nov 04 03:50:27.446787 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:27.446710 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e7a9993d-e950-4dd4-af7c-4333701ffb24/prometheus/0.log" Nov 04 03:50:27.864548 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:27.864515 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e7a9993d-e950-4dd4-af7c-4333701ffb24/config-reloader/0.log" Nov 04 03:50:28.292785 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:28.292753 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e7a9993d-e950-4dd4-af7c-4333701ffb24/thanos-sidecar/0.log" Nov 04 03:50:28.711736 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:28.711671 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e7a9993d-e950-4dd4-af7c-4333701ffb24/kube-rbac-proxy-web/0.log" Nov 04 03:50:29.132818 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:29.132790 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e7a9993d-e950-4dd4-af7c-4333701ffb24/kube-rbac-proxy/0.log" Nov 04 03:50:29.555806 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:29.555777 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e7a9993d-e950-4dd4-af7c-4333701ffb24/kube-rbac-proxy-thanos/0.log" Nov 04 03:50:29.974198 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:29.974101 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e7a9993d-e950-4dd4-af7c-4333701ffb24/init-config-reloader/0.log" Nov 04 03:50:30.606467 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:30.606434 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-874888cd5-l4bhw_a67b383f-8c42-4aca-a45c-2666a066a80e/authorino/0.log" Nov 04 03:50:30.719443 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:30.719415 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-xzvqn_eef0a147-2a4f-48fe-bc09-c96b8d05e888/manager/0.log" Nov 04 03:50:31.790257 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:31.790222 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-745f9d6695-rlnsq_4ab83526-361f-41de-b480-79fc45d7373a/telemeter-client/0.log" Nov 04 03:50:32.209752 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:32.209652 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-dcl2t_0c122db7-3d8a-4aec-b046-8e02811c7082/manager/0.log" Nov 04 03:50:32.224804 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:32.224777 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-745f9d6695-rlnsq_4ab83526-361f-41de-b480-79fc45d7373a/reload/0.log" Nov 04 03:50:32.705123 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:32.705078 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-745f9d6695-rlnsq_4ab83526-361f-41de-b480-79fc45d7373a/kube-rbac-proxy/0.log" Nov 04 03:50:33.147960 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:33.147917 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c4874f885-hqvmr_d6ebb345-a666-4ece-9afe-c1be0a44867b/thanos-query/0.log" Nov 04 03:50:33.568014 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:33.567961 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c4874f885-hqvmr_d6ebb345-a666-4ece-9afe-c1be0a44867b/kube-rbac-proxy-web/0.log" Nov 04 03:50:33.986751 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:33.986675 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c4874f885-hqvmr_d6ebb345-a666-4ece-9afe-c1be0a44867b/kube-rbac-proxy/0.log" Nov 04 03:50:34.405282 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:34.405257 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c4874f885-hqvmr_d6ebb345-a666-4ece-9afe-c1be0a44867b/prom-label-proxy/0.log" Nov 04 03:50:34.824182 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:34.824152 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c4874f885-hqvmr_d6ebb345-a666-4ece-9afe-c1be0a44867b/kube-rbac-proxy-rules/0.log" Nov 04 03:50:35.252923 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:35.252884 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c4874f885-hqvmr_d6ebb345-a666-4ece-9afe-c1be0a44867b/kube-rbac-proxy-metrics/0.log" Nov 04 03:50:37.089791 ip-10-0-133-6 kubenswrapper[5953]: I1104 03:50:37.089755 5953 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d77786fb9-9gmhr_b7f96f38-4540-4d54-a792-35996a8767c8/kube-auth-proxy/0.log"